Flexible pipeline architecture

Tags
Cross-channel platform
Agile Software Development
Digital Transformation
Microservice Architecture
Software Development

Flexible pipeline architecture improves item processing efficiency at toom

toom operates around 300 markets and an online shop with a cross-channel approach that integrates features such as Click&Collect. Efficient processing of product master data is essential for a seamless shopping experience. The rapid growth of the platform highlighted the need for a customized software solution. Working with toom, we developed a flexible, event-driven pipeline architecture that accelerates and enhances data processing. This solution improves performance and refines the rendering of complex product variants, providing a solid foundation for future developments in the online store.

Key Facts
// Flexible, event-driven pipeline architecture
// Drastically reduced item data import times
// Modular architecture ensures long-term adaptability
// Customized software solution
Technologien
// Event-driven pipeline architecture
// Kafka
// Microservices
// Product Information Management (PIM) system
Methoden
// Data replication
// Data aggregation and transformation
// Modular structure
// System-wide testing

Flexible pipeline architecture improves item processing efficiency at toom

The hardware store chain toom has around 300 locations across Germany and an award-winning online shop. For years, it has implemented a cross-channel approach with features such as Click&Reserve and Click&Collect. The processing and enrichment of product master data is essential to provide customers with a consistent shopping experience.

Project background

The architecture of the Toom online store is based on a combination of custom-built microservices and secure third-party software. Standard software offers many advantages, especially in the early stages of development. It can be deployed quickly, is well tested and allows developers to focus on developing key functionalities. The processing of product master data at toom also follows this approach. It is regularly replicated from the Product Information Management (PIM) system to the shop software and exported to connected services for further processing.

old Process toom

The rapid growth of the platform placed new demands on the existing software. Additional features required adjustments to the data models in both the PIM and shop systems. At the same time, the volume of data being processed was increasing and numerous internal services required efficient access. A bespoke software solution for toom proved to be the best choice. This solution was designed and successfully implemented by experts from Qvest Digital AG, in close cooperation with Toom's developers.

Torsten Hamper | CTO Cross Channel, toom Baumarkt GmbH

When we began the Cross-Channel project in 2016, we were introduced to Qvest Digital AG through a recommendation from our sister company, REWE digital. Since then, our collaboration with Qvest Digital has been characterized by deep trust, commitment, and quality. The team led by Kai Ebenrett takes pleasure in exploring new paths, thinking 'out of the box,' and has accompanied us from the outset on the journey to becoming the best Cross-Channel DIY retailer in Germany. My colleagues and I look forward to the upcoming years of collaboration and the exciting new topics that lie ahead.

Our solution

The development utilizes a modern, event-driven pipeline architecture. Kafka, serving as the event broker, buffers each intermediate step, allowing consuming services to respond instantly to relevant changes. The pipeline is composed of four distinct layers, each managed by its own microservice:

  1. Import: Replicates data from the PIM system
  2. Transformation: Handles data aggregation and formatting
  3. Export: Transfers data to the shop system
  4. Post-processing/Validation: Monitors the processing and import of calculated data within the shop system.

As a positive outcome, the performance of importing item data has drastically improved. What once took up to 30 minutes for individual updates now completes in just a few minutes. By prioritizing flexibility and scalability in the architecture, we've established a strong foundation for the shop’s future development. Additionally, complex product variants were mapped for the first time, leading to a clearer and more user-friendly presentation.

Challenges and Opportunities

The transformation layer should map and extend the functional logic of the previously used third-party solution. However, this business logic was incompletely documented at the start of the project. Our agile approach allowed us to quickly identify and implement the business requirements. A modular structure and continuous refactoring helped to keep the architecture lean and maintainable.

The system-wide testing of the solution was challenging in terms of communication. Thanks to the commitment and cooperation of all internal and external teams, this was completed within two weeks, followed by a successful go-live with a subsequent hypercare phase.

Conclusion

The modular, event-driven architecture not only meets all stability and performance standards but also ensures long-term adaptability. This flexible, forward-thinking solution enables seamless future updates to the system architecture. By decoupling services, outdated components can be replaced with ease and efficiency. Additionally, the collaborative effort reinforces our client's position as a leading cross-channel online retailer in Germany.

Kai Ebenrett | Chief Business Development Officer, Qvest Digital

Together with toom Baumarkt GmbH as our client, we implemented a complex IT project in an extremely agile and highly professional environment within a very short timeframe. Despite intense pressure on the project, the collaboration was very efficient and goal-oriented.

Contact
Let's talk
Get in touch with our experts.