Migrating thousands of datasets across a sprawling ecosystem is one of the toughest infrastructure challenges. At Spotify, we tackled this by introducing Background Coding Agents — automated workers that handle code generation, validation, and rollout in the background. Coupled with our internal tools Honk (orchestration), Backstage (developer portal), and Fleet Management, we turned a painful manual process into a smooth, self‑service operation. Here are five key things you need to know about this approach.

1. Automated Code Generation Eliminates Human Error

Manually writing migration scripts for thousands of datasets is error‑prone and slow. Background Coding Agents automatically generate the necessary code to transform and move datasets from source to target. They analyze schema differences, apply transformation rules, and produce validated migration scripts. This cuts out the tedious work and ensures every dataset receives consistent, correct code — no typos, no missed fields. The agents run quietly in the background, freeing engineers to focus on higher‑value tasks.

2. Honk Orchestrates the Workflow from Start to Finish

Honk is the backbone of our migration pipeline. It receives requests from developers, dispatches the appropriate coding agents, monitors their progress, and handles retries and failure on, all within a single workflow. Because Honk is event‑driven, it can coordinate complex multi‑step migrations — e.g., schema migration, data copy, validation, and rollback — without manual intervention. The result is a predictable, auditable process that scales linearly with the number of datasets.

3. Backstage Provides Self‑Service Control and Visibility

Developers interact with the migration system through Backstage, Spotify’s internal developer portal. They can trigger a migration with a few clicks, see real‑time status dashboards, and drill into logs for any dataset. Backstage also surfaces key metadata — ownership, dependencies, and migration history — so teams can make informed decisions. This self‑service interface reduces the need for central operations teams and puts power directly in the hands of the developers who own the data.

4. Fleet Management Ensures Reliable Agent Execution

Running thousands of agents concurrently requires robust fleet management. Spotify’s Fleet Management layer monitors agent health, automatically restarts failed agents, and balances the load across available compute nodes. It also handles gradual rollouts: new agent versions are tested on a small subset of migrations before being rolled out broadly. This safety net ensures that even during peak migration periods, the system stays stable and no dataset is left in an inconsistent state.

5. Downstream Consumers Stay Unaffected During Migrations

A major goal was to make migrations transparent to downstream datasets that consume the migrated data. Background Coding Agents generate compatibility layers — temporary views or transformation logic — that ensure consuming pipelines continue to work even while the underlying dataset is being moved. Once the migration is verified, the agent cleans up these compatibility layers. This “zero‑impact” approach means teams can migrate at their own pace without coordinating with every downstream team.

Conclusion
By combining Background Coding Agents with Honk, Backstage, and Fleet Management, Spotify turned a high‑risk, manual chore into an automated, self‑service capability. The result is faster migrations, fewer incidents, and happier engineers. If your organisation struggles with large‑scale dataset migrations, consider investing in a similar agent‑based approach — it might just be the supercharger you need.