interop.io / Finsemble
Diagnosed and fixed an exponential config-sync bug, cutting startup time from 181s to 20s.
Senior Software / Solutions Engineer · Nov 2020 – Sep 2023
Overview
Finsemble is interop.io's desktop interoperability platform: a distributed system of services and apps, each running in its own context, kept in sync by a large, hierarchically structured configuration that every service held a copy of and could subscribe to at any level. A large financial institution client ran into trouble as it scaled up the number of apps it registered at runtime: past a few hundred apps, startup got dramatically slower, and around 400 apps the system crashed outright trying to start.
My Role & Approach
I instrumented the process (there was almost no logging, itself part of the problem) and measured how insertion time scaled with app count. It wasn't linear, it was exponential, and the config service's own design explained why: every change was applied and notified individually, and because listeners existed at every level of the config tree, one change could fan out into notifications at every level above it. Inserting 800 apps meant 800 separate updates converging on the same upper branches of the tree, flooding the messaging layer. I reworked how config changes were applied, batching insertions so that a run of app registrations collapsed into a single notification at the parent level instead of one per app: 800 notifications became 1. Along the way I added the logging that had been missing and cleaner API signatures for config insertion.
Outcome
Startup time in large desktop environments dropped from 181 seconds to 20 seconds, and insertion time now scales linearly with app count instead of exponentially, so the crash at scale is gone. I also represented interop.io in industry standards work, co-authoring the FDC3 Agent Bridging specification.
Tech Stack
Interested in something similar?
Let's talk