The US client & redesign of data flow with DataBricks

Feb 22, 2026 Data Advisory

The client is a U.S.-based insurance provider with a strong presence across several Midwestern states. It is widely regarded as a regional market leader and is among the largest insurers serving rural communities, offering a broad portfolio of home, farm, and auto insurance products.

The project scope focuses on redesigning and implementing the CDA-to-EDW pipeline using Databricks and Azure Data Factory, with flexibility to address emerging high-priority items as directed and prioritized by the client's team.

Challenge: Guidewire Cloud migration exposed EDW data quality issues

During the data migration from the legacy platform to Guidewire Cloud, significant issues emerged with the Enterprise Data Warehouse (EDW), which was delivering inaccurate, incomplete, and unreliable data, effectively limiting its use for analytics and reporting.

Key issues included:

inconsistency and poor structure of the post‑migration data model compared with the legacy system.
overly complex, illogical, and in some cases incorrect data transformations, including missing key data elements
inability to perform most required analyses and reporting using the current data, leading to reliance on outdated pre‑migration datasets
lack of synchronization across EDW environments, resulting in bottlenecks and unnecessary dependencies that hinder efficient data enhancement and make routine development costly and error-prone

Solution: Databricks + Azure Data Factory CDA-to-EDW pipeline redesign

Sollers implemented a streamlined, consistent CDA-to-EDW data flow, precisely tailored to the client’s requirements, ensuring complete alignment between business objectives and technology architecture.

Optimized Azure environment – cleaned up and restructured to reduce costs and improve maintainability.
Standardized development setup – consistent environments, access management, and best practices established.
Scalable data ingestion process – Databricks-based solution enabling reliable and performant data ingestion from Guidewire CDA, leveraging the metadata files for processing and validation.
Azure Data Factory integration – serving as the orchestration layer for Databricks workflows, managing pipeline execution, dependencies, and scheduling across the ELT process.
Medallion Architecture implementation – standardized Bronze, Silver and Gold layers ensuring transparency and traceability of data flows.
Data validation & quality checks – automated checks ensuring accuracy, completeness, and consistency of datasets.
Comprehensive technical documentation – covering architecture, implementation, and operational procedures.

Why it works: Medallion architecture, data lineage, and automated data validation

By implementing a clear and streamlined Medallion architecture in Databricks, we eliminated complex and undocumented data transformations that previously led to unreliable outcomes. We established a framework focused on enforcing the simplicity and integrity of the Guidewire data model. As part of the deployment process, we introduced robust data validation rules and automated data quality monitoring to maintain high data standards and mitigate risk. Additionally, we implemented automated monitoring of data flows and usage, along with a rollback mechanism that enables rapid data restoration in the event of unexpected system disruptions. The Azure environments are fully synchronized, making data validation, system maintenance, and deployment processes significantly more efficient and faster. The overall architecture has been optimized for maximum efficiency with minimal resource consumption, resulting in reduced operational costs and lower maintenance effort.

Results: Trusted Guidewire Cloud data warehouse for analytics and reporting

Delivery of transparent, reliable, and high-quality data that enables the effective execution of all required analytics and reporting activities.
Promotion of the Guidewire data model, enabling clear data lineage and full alignment between business and technical stakeholders.
Efficient system performance monitoring and data quality maintenance, supporting risk mitigation and informed decision-making.
Enhanced system resilience with the ability to recover quickly during disruptions.
Accelerated validation, maintenance, and deployment processes.
Optimized overall architecture for maximum efficiency and low resource consumption, resulting in reduced operational costs and maintenance effort through architectural simplification.

Next steps: Scaling the Guidewire CDA-to-EDW pipeline for new use cases

As the focus of the Databricks design is on simplicity and effectiveness, the data model remains well-structured, familiar, and closely aligned with the source systems. The solution is designed with scalability in mind, ensuring readiness for future growth and evolving data transformation requirements driven by business needs.

Through clear documentation, active client engagement throughout the design and development process, comprehensive code reviews, and additional knowledge transfer sessions, the client is fully equipped to independently enhance and extend the system. This includes the ability to manage and modify database configurations in line with future requirements.

Conclusion

By modernizing the client’s data architecture with a streamlined Databricks driven pipeline, Sollers delivered a reliable, high-performance foundation that restores confidence in enterprise data and accelerates business insight. The new CDA to EDW flow simplifies operations, strengthens data quality, and reduces costs – empowering the organization to fully leverage Guidewire Cloud and drive smarter, faster data analytics and decision-making. This transformation positions the client for continued innovation and long-term growth in a rapidly evolving insurance landscape.