Case study 4: Measuring Experience and Key Results

What We Did Together

Built the Infrastructure from Zero

• Hired and deployed behavioral analytics (Fullstory)
  We didn't have any behavioral analytics when I started. I made the case to leadership, selected Fullstory as our platform, and worked with engineers to install it across every product. This wasn't just a plug-and-play situation—it required collaboration with engineering teams who were already stretched thin, prioritization conversations about implementation timelines, and education about why this mattered.

• Hired and trained a Quantitative UX Researcher
  I created our first dedicated quantitative research role and hired someone who could own this work strategically, not just execute surveys. Together, we built the practice: defining what metrics mattered, how to measure them, and how to translate findings into action.

• Redesigned CSAT surveys for actionable insights
  We didn't just increase sample size (from 55 to 1,100+ responses)—we redesigned the surveys entirely. Better targeting, clearer questions, and analysis that could tell teams what was broken and where to focus. We also integrated AI-powered text analysis in Qualtrics to handle open-ended responses at scale, turning hundreds of comments into thematic insights.

Made Measurement Part of How Teams Work

• Created tracking dashboards aligned with product OKRs
  For each product and major functionality, we built dashboards that tracked the metrics teams actually cared about: task completion, error rates, feature adoption, user satisfaction. These weren't generic reports—they were tailored to each team's objectives and updated continuously.

• Trained PMs on defining and tracking metrics
  We ran workshops teaching product managers how to define basic metrics, what "good" looks looked like, and when to ask research for help with deeper analysis. The goal wasn't to make everyone a quantitative researcher—it was to make measurement literacy a shared capability.

• Established rituals for measurement reporting
  We created quarterly review sessions where we presented insights to Product Leadership and the Senior Leadership Team (SLT). This wasn't just about sharing data—it was about making measurement visible at the executive level, reinforcing that this work matters.

• Required metrics definition before new product launches
  New products now can't launch without defining success metrics upfront. This simple gate forced teams to think about outcomes, not just outputs. What does success look like? How will we know if this worked? These questions became standard, not optional.

Built Cross-Functional Collaboration

Partnered with Data Analytics and CX teams
We created synergies with data analytics (who owned business metrics) and Customer Experience teams (who had support ticket insights). Instead of working in silos, we connected behavioral data with business outcomes and qualitative customer feedback. This gave us the full picture: what users do (behavioral analytics), how they feel (CSAT), why they struggle (qualitative research), and business impact (data analytics).

Collaborated across the organization
Making measurement stick required buy-in from everyone:

• Engineers to install survey pop-ups, implement tracking, and use the same tools we used

• Program Managers to give us slots in demos and "Measure & Learn" events

• Product Managers to define OKRs and create space for follow-up qualitative research when we found issues

• Designers to think about metrics during the design process and include them in "definition of done" for Jira tasks

• Data Analysts to connect behavioral metrics with business metrics in their reports

The hard parts

Changing Culture is Slow and Messy

This wasn't a six-month project with a clean ending. It was years of incremental progress, setbacks, and persistence. Early on, teams saw measurement as "extra work" that slowed them down. We had to prove value repeatedly: showing how metrics helped them prioritize, avoid rework, and celebrate wins when things improved.

Different Teams, Different Maturity Levels

Some teams embraced measurement immediately. Others resisted for months. We couldn't force cultural change—we had to meet teams where they were, demonstrate value on their terms, and celebrate early adopters who became internal advocates.

The "So What?" Problem

Early dashboards showed data but didn't always drive action. We learned that numbers alone don't change behavior: you need clear recommendations, prioritized next steps, and follow-up to ensure insights become action.

Technical Complexity

Installing Fullstory across legacy products wasn't straightforward. Some products had technical constraints. Some teams had competing priorities. We had to negotiate, compromise, and phase rollouts strategically.

Requires Endless Collaboration

Every step needed collaborators. Want to add a survey pop-up? Need engineering time. Want metrics in OKRs? Need PM alignment. Want designers to care? Need to show how metrics validate their work. Building measurement culture is as much relationship-building as it is methodology.

What I Learned 

Culture Change Doesn't Happen Overnight

We started this work in 2022. We're still refining it in 2025. Different teams have different maturity levels. That's okay. Sustainable change happens through persistent, incremental wins—not grand transformations.

Make Allies, Not Requirements

Early on, we tried mandating measurement. It didn't work. What worked was finding early adopters, proving value on their products, and letting their success stories convince skeptical teams. Internal advocates are more powerful than top-down mandates.

Show the "Why," Not Just the "What"

Teams don't care about CSAT scores abstractly. They care when you show them why the score is low (error messages confuse users) and what they can do about it (here are three quick fixes). Connect metrics to action, always.

Rituals Matter More Than Tools

Fullstory and Qualtrics are powerful, but they don't change culture by existing. What changed culture was quarterly reviews, demo presentations, Slack posts celebrating improvements, and workshops where teams practiced defining metrics. Tools enable measurement; rituals embed it.

"Perfect" is the Enemy of "Done"

We didn't wait for perfect dashboards or complete behavioral tracking before reporting insights. We started with what we had, improved iteratively, and demonstrated value early. Waiting for perfect would have meant never starting.

Measurement Empowers, It Doesn't Threaten

Some teams initially feared measurement would expose failures. We reframed it: measurement shows where to improve, celebrates wins, and protects teams from building the wrong things. When teams saw metrics as their tool (not a judgment on their work), adoption accelerated.

Companies Are Delivery Factories Until They Choose Not to Be

Organizations default to shipping features because it feels like progress. Measurement and iteration feel slower, harder, less tangible. Changing that requires leadership commitment, not just research enthusiasm. We succeeded because executives eventually saw measurement as strategic, not operational.