Skip to main content
A/B testing compares your current search configuration (the control) with one or more alternatives (the variants) using live traffic. Use analytics to measure whether relevance tuning improves your search results.

​
Advantages of A/B testing

A/B testing helps you:
  • Limit disruption: test changes to an or search configuration with a share of your live traffic before applying them to all users.
  • Base decisions on user behavior: use click and conversion events to compare how configurations affect engagement.

​
Set up A/B tests

After you configure your app to send click and conversion events, you can create and manage A/B tests in the Algolia dashboard without writing more code.

​
Collect click and conversion events

Configure your app to send click and conversion events before you start a test. These events let you compare clicks and conversions for each variant.

​
Set up indices or search parameters

The setup depends on whether you want to compare index settings or search parameters: For more information, see Create and launch a test.

​
Run A/B tests

After you prepare your indices or search parameters, create and launch an A/B test. Review results using the statistical method you selected. For frequentist tests, wait until the planned end before making a decision. For Bayesian tests, see Bayesian experimentation. Use what you learn to plan your next test.

​
A/B testing in the Algolia dashboard

In the Algolia dashboard, select your Algolia application. On the left sidebar, select Search, then A/B Testing. The overview page lists each test’s status, start date, and duration. Select a test to see the metrics for each variant. Click View analytics to open search analytics for the test’s variants.

​
Example A/B tests

The following examples compare custom ranking, searchable attributes, and rules. You can also test other search-time settings, such as typo tolerance and optional filters.

​
Test custom ranking by number of likes

Your users can like items such as music, films, and blog posts. To test whether custom ranking by number of likes improves your search results:
  1. On the A/B Testing page, click Create test and select Index vs. Index test. Select the main index as the control and the replica as variant B.
  2. Add a numeric number_of_likes attribute to the in your main catalog index.
  3. In the replica’s settings, add desc(number_of_likes) to customRanking. Keep the other settings unchanged.
  4. Name your test “Test new ranking with number of likes”.
  5. Assign 10% of traffic to variant B and 90% to the control to limit how many users see the ranking change.
  6. If the sample size estimator is available for this test type, use it to choose a duration. Otherwise, set the duration based on your traffic and the effect you want to detect.
  7. Name the test Test ranking by number of likes. Review the configuration and click Launch test.
  8. Review the results using the test’s statistical method. Compare any improvement with the cost of changing the ranking before adopting it.

​
Test a searchable short description

Your records include a short_description attribute. This example assumes your main index has an explicit searchableAttributes list that doesn’t include it. To test whether searching this attribute improves relevance, follow these steps:
  1. Use your main index as the control. Don’t change its searchableAttributes setting.
  2. Create a standard replica for variant B. Add short_description to the end of the replica’s searchableAttributes list. Keep the other settings unchanged.
  3. On the A/B Testing page, click Create test and select Index vs. Index test. Select the main index as the control and the replica as variant B.
  4. Assign 30% of traffic to variant B and 70% to the control to limit how many users search the added attribute.
  5. If the sample size estimator is available for this test type, use it to choose a duration. Otherwise, set the duration based on your traffic and the effect you want to detect.
  6. Name the test Test searchable short descriptions. Review the configuration and click Launch test.
  7. Review the results using the test’s statistical method. Compare any improvement with the cost of making short_description searchable.

​
Compare searches with and without rules

Use this test to compare searches that apply index rules with searches that don’t. This measures the combined effect of the rules, not just the effect of the promotion in this example. In this example, you promote an iPhone model for queries containing apple, iphone, or mobile. To compare the effect of rules on clicks and conversions, follow these steps:
  1. In your main index, set enableRules to true. Use this index as the control.
  2. Create a rule on the main index that promotes the iPhone record to the first position for the target queries.
  3. On the A/B Testing page, click Create test and select Index configurations. Use the main index for the control and variant B.
  4. For variant B, click Add Query Parameter. Next to Enable Rules, turn off Tested value.
  5. Assign 30% of traffic to variant B and 70% to the control to limit how many users search without rules.
  6. If the sample size estimator is available for this test type, use it to choose a duration. Otherwise, set the duration based on your traffic and the effect you want to detect.
  7. Name the test Test searches with and without rules. Review the configuration and click Launch test.
  8. Review the results using the test’s statistical method. Compare clicks and conversions between the control and variant B to assess the effect of rules.

​
See also

Last modified on September 23, 2026