What Really Happens at a Cleo Backend Chapter Day?

1. What is Backend Chapter Day?

Here at Cleo, our Backend Chapter have regular meetings where we work together to tackle the gnarliest problems, share our successes, and keep Cleo's infrastructure humming. Twice a year we go further than that, gathering in person for our Backend Chapter Day.

Backend Chapter Day brings us together and gives us a chance to surface from the depths of database optimization and queue management to learn from each other's toughest challenges. Expect deep technical discussions, hard-won insights, and, inevitably, passionate debates that continue long after the pizza arrives.

2. Chapter Day Content

Bingo & Welcomes

Sachin got everyone settled with Backend Bingo - surfacing some surprising facts about our colleagues and giving us all a chance to catch up with team members, old and new.

Scene Setting: Productivity Through Better Tooling

Murray tackled a question that hits every engineer daily: how do we make our work more efficient through better tooling? The session helped frame the day's theme—we're not just building features, we're building the systems that help us build better features.

Lightning Talks: The Technical Deep Dives ⚡

AI Test Suite

David's talk tackled the challenge of testing LLM-powered features. The core problem? Traditional deterministic testing breaks down when you're working with AI that learns and adapts.

His slides showed how he and his team have built a framework that embraces the reality that "everything is flaky by default" — so instead of expecting exact outputs, the system measures overall success percentages.

Sidekiq Deep Dive

Edward walked us through Sidekiq's architecture using an experiment run by the Payments Infrastructure team as his vehicle—a drive designed to reduce missed EWA (Early Wage Access) repayments by predicting when users' income would land.

What started as a straightforward "run payment attempts at 23:00 UTC" quickly revealed the complexity lurking beneath Sidekiq's surface. The session took us on a deep dive into the guts of Sidekiq, following the complete journey from client to Redis to server, breaking down how jobs get normalized, serialized, pushed to queues, and eventually processed by worker threads.

Audio in Ruby

Lily's talk was a deep dive into digital audio fundamentals and how we handle voice at Cleo. She started with the basics—explaining how digital audio works through sampling rates, bit depth, and channels—before showing the practical differences between WAV and MP3 formats.

Her talk revealed the technical complexity behind our voice features. For voice notes, we receive raw binary blobs from ElevenLabs over HTTP. For Talk Mode (our two-way voice feature), we get base64-encoded audio chunks over WebSockets that need to be concatenated in real-time.

Workshop: Cleo Instruments & Dashboards

Chris led us through our observability framework—because when you're handling real money, you need to know when things break before your users do. We split into cross-team groups to explore existing metrics and identify opportunities for adding new instrumentation. The workshop covered:

• How to add meaningful metrics without exploding cardinality

• Building dashboards that actually help during incidents

• Setting up alerts that inform rather than overwhelm

Key takeaway: engineers should be the first to know when something's wrong, not the last.

Midday Fuel & Conversations 🥪

An excellent lunch spread gave us fuel for the afternoon, but more importantly, created those spontaneous moments where complex problems get solved. Nothing quite like a casual lunch conversation to unlock that architectural insight you've been wrestling with all week.

Quiz Refresher & Afternoon Sessions

Post-lunch, Sachin brought us back with a fun stats quiz about our codebase: "How many lines of code have been added since the last Backend day?" The answer revealed just how much we'd shipped as a chapter.

Workshop: Live Refactor

This was hands-on engineering at its finest. Gavin and Alex took a real piece of production code and refactored it live.

The beauty of watching code evolve in real-time? You get to see how other engineers actually think—what they notice first, which trade-offs they consider, and how they balance competing priorities. It's the kind of learning that only happens when you can peer over someone's shoulder as they work.

More Lightning Talks ⚡

Perception and Productivity

Matt delivered a thought-provoking talk on cognitive biases in engineering work, drawing on Daniel Kahneman's research around System 1 (automatic, intuitive) vs System 2 (deliberate, analytical) thinking. His core thesis: small inefficiencies compound dramatically over time.

Walking through concepts including habituation (you stop noticing things after a while), systems thinking, and the cargo cult fallacy—where we assume correlation equals causation- Matt highlighted how small changes in approach and thinking can produce an outsized impact on our productivity.

Performance Deep Dive

Rob shared the epic journey of optimizing our prompt framework—a system that decides which notifications and messages users see. What started as "it works!" in February 2024 became a months-long performance overhaul involving:

• High-level caching for boolean audience checks

• Optimized data fetching to reduce database queries

• Strategic ordering of expensive operations

• Custom instrumentation for granular metrics

The results? Guaranteed reasonable response times and a system that won't degrade over time as we add more prompts.

Chat 3.0 Tools

Sean closed our technical sessions with a deep dive into the tools that power our AI chat system. These aren't just simple API calls—they're sophisticated service objects that bridge the gap between our LLM agent and Cleo's backend systems. The challenge? The agent can be "wacky," so tools need robust error handling, clear descriptions, and fallback responses.

Social & Wrap-Up 🍕

That wrapped up our formal sessions, but the day's real value emerged in the hours that followed. Over pizza and drinks, conversations ranged from architectural philosophy to debugging war stories. The kind of deep technical discussions that only happen when passionate engineers gather with adequate refreshments.

3. Beyond the Day: Building Community Across Time Zones

Backend Chapter Day reaches every corner of our distributed team:

• Remote engineers join us in person where possible, with travel and accommodation fully covered.

• Cross-team collaboration flows naturally, with engineers from across the chapter contributing their perspectives.

• Knowledge sharing extends well beyond the official agenda, with conversations and informal mentoring continuing the learning

4. What Makes Backend Chapter Day Special?

Backend Chapter Day perfectly embodies Cleo's engineering principles in action:

• We build software that delivers value: Every session focuses on real user problems and practical solutions that improve our product and user experience

• We make small changes that deliver business value quickly: The day showcases iterative improvements and incremental optimizations that compound into significant impact over time

• We choose technologies that allow us to make an impact: Technical decisions are always grounded in clear purpose—making our systems more reliable, efficient, and effective for users

• We give each other useful, constructive feedback: The workshops and discussions create space for collaborative learning and knowledge sharing across the entire chapter

• If someone is stuck, helping them is the highest-value use of our time: The entire day is structured around this principle—spreading knowledge and levelling everyone up together

Backend Chapter Day isn't just a showcase of technical achievements—it's our engineering principles made tangible through community learning and shared problem-solving.

5. Why This Matters for Engineers Considering Cleo

If you're thinking about joining Cleo as a backend engineer, here's what Backend Chapter Day reveals about life here:

• We tackle genuinely complex problems: From LLM testing frameworks to sub-second prompt selection systems, you'll work on challenges that push the boundaries of what's possible

• We measure what matters: Our observability culture means you'll ship with confidence, knowing exactly how your code performs in production

• We're hybrid-first and remote-inclusive: Whether you're optimizing queries from London or debugging workers from Lisbon, you're fully part of the team

• We balance technical excellence with business impact: Every optimization and architecture decision connects back to better user experiences and business outcomes

P.S. – We're hiring! Engineering roles are open now → Check them out here