> ## Documentation Index
> Fetch the complete documentation index at: https://toolathlon.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# A/B Testing

> Analyze the homepage A/B test in BigQuery, complete record.csv, determine the winner by mean per-scenario conversion rate, and either create the promotion bucket for B or write the required no-change log for A or a tie.

> **Toolathlon-Verified** · Website task ID `141` · Canonical task `ab-testing`
>
> [View the task source at `d57361c0`](https://github.com/hkust-nlp/Toolathlon/tree/d57361c0f1582cf9a0675c0753315bb6b004bd0e/tasks/finalpool/ab-testing)

## Required Tools

<Card>
  <div className="tools-container">
    <div className="mcp-servers-container">
      <div className="mcp-servers-title">
        MCP Servers
      </div>

      <div className="mcp-servers-grid">
        <div className="mcp-server-item">
          <img alt="" aria-hidden="true" src="https://mintcdn.com/hkust/QdN2Y6f6c_GBV7Vj/icons/google_cloud.png?fit=max&auto=format&n=QdN2Y6f6c_GBV7Vj&q=85&s=cf7e733dfef17cb79e44c97048c2f8e5" style={{height: "20px", width: "20px", margin: 0, padding: 0, display: 'inline-block'}} width="64" height="64" data-path="icons/google_cloud.png" />

          <span className="mcp-server-name">google-cloud</span>
        </div>

        <div className="mcp-server-item">
          <img alt="" aria-hidden="true" src="https://mintcdn.com/hkust/QdN2Y6f6c_GBV7Vj/icons/filesystem.png?fit=max&auto=format&n=QdN2Y6f6c_GBV7Vj&q=85&s=b4163322f58a72b712c697bdbeac85b9" style={{height: "20px", width: "20px", margin: 0, padding: 0, display: 'inline-block'}} width="64" height="64" data-path="icons/filesystem.png" />

          <span className="mcp-server-name">filesystem</span>
        </div>
      </div>
    </div>

    <div className="local-tools-container">
      <div className="mcp-servers-title">
        Local Tools
      </div>

      <div className="local-tools-grid">
        <div className="local-tool-item">
          <img alt="" aria-hidden="true" src="https://mintcdn.com/hkust/ucpMq24upueB4g4R/icons/claim_done.png?fit=max&auto=format&n=ucpMq24upueB4g4R&q=85&s=94ee5a6cf860bae09936a702c6c6c648" style={{height: "20px", width: "20px", margin: 0, padding: 0, display: 'inline-block'}} width="14" height="14" data-path="icons/claim_done.png" />

          <span className="local-tool-name">claim\_done</span>
        </div>

        <div className="local-tool-item">
          <img alt="" aria-hidden="true" src="https://mintcdn.com/hkust/v2Ad3GlVGK-9gZ_R/icons/python.png?fit=max&auto=format&n=v2Ad3GlVGK-9gZ_R&q=85&s=6e4e3cca411e7f15013ce15afa0dd155" style={{height: "20px", width: "20px", margin: 0, padding: 0, display: 'inline-block'}} width="64" height="64" data-path="icons/python.png" />

          <span className="local-tool-name">python\_execute</span>
        </div>

        <div className="local-tool-item">
          <Icon icon="filter-list" size={20} color="#4286f6" />

          <span className="local-tool-name">handle\_overlong\_tool\_outputs</span>
        </div>

        <div className="local-tool-item">
          <Icon icon="bars-progress" size={20} color="#4286f6" />

          <span className="local-tool-name">manage\_context</span>
        </div>

        <div className="local-tool-item">
          <img alt="" aria-hidden="true" src="https://mintcdn.com/hkust/ucpMq24upueB4g4R/icons/history.png?fit=max&auto=format&n=ucpMq24upueB4g4R&q=85&s=25d885756cf14ced8475bd94adfdecb7" style={{height: "20px", width: "20px", margin: 0, padding: 0, display: 'inline-block'}} width="14" height="14" data-path="icons/history.png" />

          <span className="local-tool-name">history</span>
        </div>
      </div>
    </div>
  </div>
</Card>

## Instruction

The A/B test for our new homepage has concluded, and the raw clickstream data has been stored in the `ab_testing` dataset in BigQuery. Analyze the data and fill the `record.csv`, determine which version ('A' or 'B') has the highest overall conversion rate, i.e., the overall conversion rate is defined as the arithmetic mean of the per-scenario conversion rates. If version B outperforms, immediately create a new Cloud Storage bucket whose name is prefixed with `promo-assets-for-b` for the full promotion, and you do not need to write any log entry in this process. If version A wins or the results are a tie, no bucket creation is required, but a log entry with the message `{'status': 'AB_Test_Concluded', 'winner': 'A', 'action': 'No_Change'}` must be written to the existing log bucket whose name is prefixed with `abtesting_logging`. During the entire process, there is no need to consider any log entries in the log bucket that existed before the task was executed.

## Initial State

### Local Workspace

<div className="file-tree">
  <a href="https://github.com/hkust-nlp/Toolathlon/tree/d57361c0f1582cf9a0675c0753315bb6b004bd0e/tasks/finalpool/ab-testing/initial_workspace">workspace</a>/
  └── record.csv
</div>

### Runtime Setup

This task initializes application or service state during preprocessing. [Review the pinned setup source](https://github.com/hkust-nlp/Toolathlon/tree/d57361c0f1582cf9a0675c0753315bb6b004bd0e/tasks/finalpool/ab-testing/preprocess).

## Legacy Trajectories

<Warning>
  These replays were produced on the original Toolathlon release. They are retained for historical inspection and are not Toolathlon-Verified results.
</Warning>

* ❌ [Claude-4.5-Sonnet](https://toolathlon-traj.xyz/claude-4.5-sonnet_ab-testing)
* ❌ [Deepseek-v3.2-Exp](https://toolathlon-traj.xyz/deepseek-v3.2-exp_ab-testing)