Reducing Salesforce Data Loading Time From 12 Hours to 41 Minutes

10 years ago, a script based approach to integration seemed like a good idea for a leading US wealth management firm.

9 years later, after very substantial growth in clients and data sources, the maintainability, error rates and lag times had become a serious threat to transparency and overall quality of service.

The company wanted to have complete control of their systems and steer clear of expensive ETL tools and inflexible platforms. This strategy served them well initially but as the business grew so did the complexity and the data volumes. The difficulties peaked as they developed a highly configurable Salesforce portal for their financial advisors.

With tens of millions of records of data being processed daily, things were out of hand. Maintaining ever-growing libraries of legacy scripts was now a serious problem and error rates had become very significant, requiring a full day of corrections. In the end, the published data  was often 2 days out of date.

At this point the company realized that they had to squarely address the problem or the ability for the business to maintain a dominant position would be jeopardized. Choosing a platform partner was not just a question of product, it was equally about the team behind the product. CloverDX stood out from the competition in this respect with our experience in the integration field and the ability to co-engineer a solution.

Reducing Salesforce data loading time and freeing up resource

With this new solution, the company has been able to vastly reduce the Salesforce data loading time and free up significant resource for the team, so they can focus on adding value - not just fixing errors. The solution now sequences data appropriately to overcome the Salesforce data loading challenges, and data validation is now robust and reliable. It’s also enabled better information to be provided to their customers with data being pulled multiple times a day from different sources.

With improved data ingest, their platform can now also integrate with legacy systems; new integrations are significantly faster; customers can be onboarded more quickly; and the business can focus on their growth.

• Needed to feed a complex Salesforce portal providing financial advisors a 360 degree view of the client
• Tens of millions of records processed daily
• Success of the new business model hinged on loading data, quickly and error free, into Salesforce
• Needed to overcome the timeouts and batch restrictions imposed by the Salesforce API
• Company needed not just product but expertise and a team to help accelerate the solution
• Move away from hard to maintain scripts
• Load times reduced from 12 hours down to 41 minutes
• Error rate reduced to virtually nothing
• Data sources due to double does not raise concern for the business
• Dev team now focuses on adding value, not fighting fires
• Can now integrate with any legacy accounting systems
• Bolt on integrations that used to take a month now take less than a week

Turning 12 hours of error prone data processing into 41 minutes of accuracy Explore Topic: Data Ingest