Production system, CRM and QuickBooks connected with Apache Camel

The challenge

A SaaS platform selling automotive data to US businesses runs its own production system covering the whole customer lifecycle: sign-up, packages, balance, invoices and payments. Sales and support, however, worked in a CRM, and the books were kept in QuickBooks. The three did not know about each other — a new customer existed in the production system but had to be retyped into the CRM, and each payment had to be booked separately.

On top of that, the company was moving from an old CRM version to a new one — an open-source CRM derived from SugarCRM. Over the years the old installation had accumulated accounts, contacts, notes, documents, products and customer licences. The team could not stop working for a one-off migration, so the old and new CRM had to run side by side for a while.

The solution

We built a single integration service on Apache Camel and Spring Boot. Every data flow is a separate route with its own schedule, error handling and a durable record of what has already been processed.

01 Diagram of the data flow between the production system, the CRM and QuickBooks

Production system → CRM. Data flows on an ongoing basis: every hour — often enough to keep the CRM current, and rarely enough not to load the production database. On each run the service picks up customers created since the previous run and creates an organisation in the CRM with a linked contact. The position of the last processed record is stored durably, so a restart neither skips nor re-imports anything. Separate hourly routes push each customer's account balance and active packages to the CRM — only values that actually changed are sent.

Production system → QuickBooks. The service exposes an API with newly paid invoices and customer details. On the accounting side runs a connector we wrote — a Windows service next to QuickBooks Desktop that talks to it through the QuickBooks SDK (QBXML). Every few hours it fetches new invoices in batches, creates the customer if they do not exist yet and records the payment as a sales receipt. Before writing, it checks whether a document with that number already exists, and it remembers the last processed invoice — so a retry never duplicates anything in the books.

Old CRM → new CRM. A continuous migration running every few hours moves accounts, contacts, notes, attachments, products and customer licences — the latter becoming service contracts in the new CRM. Records changed in the old system after migration can be re-synchronised; the service compares modification dates and never overwrites newer data.

Technology — what makes it work on bad days

• Idempotent consumers. Each flow keeps a durable registry of processed records. Re-running a flow — after a failure, a restart or a manual trigger — creates no duplicates. For balances and packages the key is "customer + value", so a record goes back to the CRM only when the value has changed.
• ID mapping. The same customer has a different ID in every system. The service maintains an old-to-new mapping table and uses it to rebuild relations — a contact lands under the right organisation, a note under the right contact, an attachment under the right document.
• Retries and business errors. Transient CRM connection errors are retried. A "duplicate" response is an expected case (the record is skipped); any other error is treated as a failure, with the full request and response logged.
• Go easy on production. Imports from the source run in small batches ordered by an indexed column, so the system serving customers is not loaded down.
• CI/CD. The Docker image is built and deployed from a GitLab CI pipeline.

The result

• A customer who signs up on the platform appears in the CRM automatically — as an organisation with a contact, balance and active packages.
• Customer payments reach QuickBooks without manual re-entry.
• The switch from the old CRM to the new one happened without interrupting the team: the migration ran in the background, and the first full production migration started less than two months after the first commit.
• The integration has been running without interruption for over 7 years. Along the way new flows (account balance, packages, note-to-contact links) were added without rebuilding what already worked.

Lessons learned

• Retrying the whole is not retrying a step. When creating a contact failed (for instance because the source record had no surname), the entire flow was retried — but the organisation created in the first step already existed, so organisations got duplicated. The fix normalised data before sending it to the CRM and handled "duplicate" explicitly as a skip. The general rule: every step of a multi-step flow must be idempotent on its own, not just the flow as a whole.
• Timestamps are not unique. Many source records shared a timestamp, so "fetch everything newer than the last one" either lost or repeated data. The answer was to combine the timestamp (where to start) with an idempotent consumer (what to reject).
• Do not forward everything. By default Camel carries message headers into outgoing HTTP calls. Some exceeded allowed lengths and broke requests to the CRM. Since then only an explicit allow-list of headers reaches external APIs.
• Production differs from test. The production CRM instance had a different custom-field setup than the test one — the field mapping became configuration rather than code.

More about our work for this platform: SaaS platform modernisation.