Spotify researchers report that a conversational music agent increased user listening by 14% in a two-week A/B test against an earlier experience that only let listeners refine a session. They also report 5% more weekly active users and a 5% lower skip rate among people interacting with the agent. The paper says the test reached about 15 million Spotify users across several markets and device types.
The newer experience lets listeners make music requests across several chat turns and ask questions about music. Spotify says it rolled out the agent after the test. The paper describes how the team prepared it before launch, when there was little real conversation data to train or evaluate against.
Building conversations before users supplied them
The researchers turned single-turn prompts into synthetic multi-turn conversations. They used those conversations to test whether the agent could follow a changed request, resolve references to earlier suggestions, and keep constraints across turns. In a separate improvement loop, a coding agent examined failures, changed prompts or tool descriptions, and sent changes for human review.
On a roughly 1,000-example offline set, one run of that improvement loop raised the agent’s pass rate by 8% relative to an already tuned manual prompt. The paper reports a pairwise sign-test result of p = 0.01. Its multi-turn evaluation also found that performance fell as conversations got longer, especially from five turns onward.
