Experimentation is essential for data-driven product development, but centralized experimentation systems often become bottlenecks, limiting innovation and velocity. At GetYourGuide, we faced this challenge and decided to democratize experimentation, enabling analysts and product teams across the company to define, run, and analyze experiments independently. In this session, we'll share practical insights from our journey toward democratization, focusing on technical implementation details and lessons learned.

From Centralized to Decentralized Experimentation
Initially, experimentation at GetYourGuide was centralized, limiting flexibility and slowing down decision-making. We recognized the need to empower individual contributors (ICs) by creating a self-service experimentation platform. We'll discuss the practical challenges we encountered, including managing complexity, maintaining consistency, and ensuring data quality across decentralized teams.

Enabling Flexible Metric and Dimension Definitions
To democratize experimentation effectively, we needed to empower analysts to define their own metrics and dimensions without heavy engineering involvement. We'll share how we designed a modular SQL-template approach, allowing analysts to quickly create, test, and deploy new definitions. We'll illustrate this approach with real-world examples, such as conversion rate, revenue per visitor, channel splits, and platform segmentation, demonstrating how this flexibility significantly accelerated experimentation velocity.

Standardizing Statistical Calculations with the Analyzer Toolkit
Our initial experimentation infrastructure relied heavily on Looker data models, which proved insufficient for complex statistical methods like sequential testing. To address this, we built a Python-based analysis package, the Analyzer, that standardized statistical calculations and provided reusable components. We'll explain how analysts leverage this toolkit to ensure consistency, accuracy, and extensibility of statistical methods. We'll also share how the Analyzer became a valuable resource beyond experimentation, supporting broader analytical use-cases across the organization.

Batch Processing and API-Driven Experiment Results
To ensure timely access to experiment results, we implemented a robust batch processing pipeline that pre-calculates daily experiment impressions, metrics, and dimensions. Additionally, we developed a flexible API layer to enable analysts to retrieve specific experiment results dynamically, without waiting for scheduled batch jobs. We'll discuss the technical architecture behind this dual approach, highlighting how it balances efficiency, reliability, and flexibility.

Key Lessons and Takeaways
Attendees will leave this session with practical insights into:
* Democratizing experimentation to accelerate innovation and velocity.
* Best practices for designing flexible, scalable, and maintainable experimentation infrastructure.
* Technical strategies for enabling self-service metric/dimension definitions, standardized statistical calculations, and extensible analytical capabilities.

We'll conclude by briefly outlining our future plans, including additional discriminators, advanced statistical methods, and further UI enhancements aimed at continuous democratization.