AB-test project

Let me ___ the system step by step.

walk you through
(to explain something carefully from beginning to end)

The main ___ came from the recommendation systems team.

business problem
(the real problem or need that the company wanted to solve)

Before our system, analysts had to ___.

analyze each experiment manually
(perform the analysis themselves without automation)

Our goal was not to replace analysts, but to ___.

help them make the final decision
(support analysts so they can reach a conclusion faster and more confidently)

The system first retrieved ___.

experiment metadata
(information describing the experiment, such as dates, groups, and target metrics)

We had a control group and ___.

a test group
(the group that receives the new treatment or change in an A/B experiment)

The system identified the experiment's ___.

target metrics
(the main metrics used to evaluate whether an experiment was successful)

We already had ___ for the whole experiment.

precomputed A/B-test results
(results that had been calculated before the agent started its analysis)

The existing platform did not provide ___.

meaningful user segmentation
(useful division of users into smaller groups with interpretable characteristics)

The main value of our system was finding ___.

narrower, interpretable user segments
(smaller groups of users that humans can easily understand and analyze)

We searched for segments with ___.

significant metric shifts
(large and important changes in the target metrics)

For segmentation, we implemented ___.

a beam-search-like algorithm
(an algorithm that keeps only the most promising candidates at each search step instead of exploring everything)

The algorithm explored ___.

the search space
(the complete set of possible candidates or solutions that an algorithm could examine)

At every step, the algorithm generated ___.

candidate segments
(possible user segments that could be selected for further analysis)

We had to ___ the most promising segments.

rank and filter
(order candidates by quality and remove candidates that are not useful)

We used beam search instead of ___.

checking every possible combination
(evaluating all possible combinations, which can become computationally expensive)

For each selected segment, we ___.

calculated the metrics ourselves
(computed the required metric values inside our own system)

Then we passed the results to ___.

the LLM
(the large language model responsible for generating explanations and hypotheses)

The LLM generated ___.

hypotheses about why the metric changed
(possible explanations for the observed change in a metric)

We generated SQL queries to ___.

validate the hypotheses
(check using real data whether an explanation is supported by evidence)

A hypothesis was shown to the analyst only if it was ___.

supported by the data
(confirmed or justified by information from the database)

We did not trust the LLM for ___.

numeric values
(numbers such as metric values, percentages, counts, or statistical results)

The LLM could sometimes ___.

hallucinate metric values
(generate numbers that were not actually present in the source data)

Hive remained our ___.

source of truth
(the authoritative system whose data is considered correct)

Before returning the response, the backend ___.

replaced the generated numbers with values from the source of truth
(substituted potentially unreliable LLM-generated numbers with verified database values)

The final result was returned as ___.

structured JSON output
(a response organized according to a predefined JSON structure)

The frontend used the JSON to ___.

render the final report
(display the analysis results to the user)

One of the most important parts of the system was ___.

guardrails
(technical restrictions designed to prevent unsafe or incorrect model behavior)

The main technical challenge was ___.

the segmentation algorithm
(the most difficult engineering part of the project)

We had to avoid making the search space ___.

too large to explore efficiently
(so big that evaluating all possible candidates would require too much computation)

The system reduced ___.

the average analysis time
(the typical amount of time analysts needed to analyze one experiment)

We reduced the average analysis time ___.

from about 44 minutes to 19 minutes
(a natural way to describe the measurable business impact of the project)

The analyst still ___.

made the final decision
(remained responsible for deciding whether the experiment result was acceptable)

The agent was designed to ___.

assist analysts rather than replace them
(help human specialists perform their work instead of completely removing them from the process)