App Store A/B Testing: How to Test Screenshots

Most people decide whether to install an app from a glance at its screenshots, often without reading the description. Opinions about what works are cheap, though, and a redesign that looks better to your team can still lose installs.

A/B testing settles the question with real store visitors, and both Apple and Google have free native tools for it. This guide covers how those tools work, what to test first, how long to run a test, how to read the result, and what to do with little traffic.

What A/B testing means on the app stores

In an app store A/B test (also called ASO split testing), the store shows your current listing (the control) to one group of visitors and a changed version (the treatment or variant) to another. Visitors are assigned at random, so the only systematic difference between the groups is your change.

That is more reliable than comparing this month to last month, because before-and-after numbers get mixed up with keyword rankings, featuring, seasonality and ad spend. A randomized test runs both versions in the same period, which removes most of that noise.

The metric that matters is conversion. On the App Store, App Analytics calculates conversion rate from impressions, which include search results and browse placements, not only full product page views. In search results, your first screenshots (or app preview) are often the only part of your listing a person sees before deciding whether to tap.

Apple: Product Page Optimization

Product Page Optimization (PPO) lives in App Store Connect. It lets you test up to three treatments against your original product page. A treatment can change the app icon, the screenshots, the app previews, or a combination. Text such as the name, subtitle and description cannot be tested this way.

How a PPO test works

• Traffic share: you choose what percentage of visitors see a treatment instead of the original, and that share is split evenly across your treatments.
• Localizations: you pick which localizations the test runs in, and you need assets for each one you include.
• App Review: every treatment is reviewed before the test starts. Treatments can be submitted on their own, without a new app version.
• Icons: an icon treatment only works if the alternate icon is included in the binary of the live app version, so plan icon tests a release ahead.
• Duration: a test can run for up to 90 days. You can stop it earlier at any time.
• Results: App Analytics shows each treatment's conversion rate, its estimated improvement over the original, and how confident Apple is in that difference. If a treatment wins, you can apply it to your original product page.

Custom product pages are not A/B tests

Custom product pages do a different job. Apple lets you create dozens of extra versions of your product page, each with its own screenshots, app previews and promotional text, and each with a unique URL. You send specific traffic to them: an Apple Ads ad variation, a social campaign, an influencer link.

Because visitors arrive from different sources, comparing custom product pages is not a randomized test. It is still useful for checking whether a message aimed at one audience converts better than your default page.

Google Play: store listing experiments

On Google Play, the equivalent tool is store listing experiments in Play Console. It covers more of the listing than Apple's: you can test the app icon, feature graphic, screenshots, promo video, short description and full description. Each experiment compares up to three variants with your current listing.

Default graphics vs localized experiments

There are two kinds. A default graphics experiment tests graphic assets on your default listing, and it reaches users who see your default graphics. A localized experiment runs in specific languages and can test both graphics and text. If most installs come from one or two markets, a localized experiment there gives the cleanest answer.

Settings that decide how long you wait

When you set up an experiment you choose:

• Audience: the percentage of store listing visitors who see a variant instead of the current listing. The share is split across variants.
• Metric: first-time installers or retained first-time installers. Retained installers is the better default: installs from people who delete the app the next day are not a real win.
• Minimum detectable effect: the smallest difference you care about detecting. The smaller it is, the more visitors the experiment needs. Play Console estimates how long that will take based on your traffic.

The feature graphic generator is a quick way to produce a feature graphic variant at the required 1024 x 500 size.

What to test first

Start with what most visitors actually see: many never swipe past the first few screenshots. For design ideas, see our guide to app store screenshot best practices. These tests usually earn their place early:

The first screenshot, then the first three

Your first screenshot does most of the work. Test what it leads with: your core benefit, your most distinctive feature, or the main screen of the app. Once the first slot is settled, test the set of the first three as a unit, since with portrait screenshots and no preview video, the first three appear together in App Store search results.

Captions

Caption copy is cheap to change. A useful first test is benefit against feature: "Find what you need, faster" against "Advanced filtering". Keep the visuals identical so the words are the only difference.

Screenshot order

Reordering costs almost nothing. If one feature drives retention, test moving its screenshot into the first or second position.

Device frames vs lifestyle imagery

Some categories do better with clean UI in a device frame, others with the app shown in context (a person cooking with a recipe app, a runner checking a watch). Test the visual style while keeping the captions and order the same.

Color and background

Look at the top apps for your main keywords. If they all share a palette, a contrasting background can stand out in a results list. Test it rather than assume it helps, since a color that clashes with your icon can also hurt.

App preview video vs no video

On the App Store, an app preview autoplays muted and takes the first position in search results, so it replaces what your first screenshot would have shown. You can add up to three previews per localization, each up to 30 seconds. Some apps gain from a preview and some lose, often because the first seconds show a logo or splash screen instead of the app in use. Testing the same screenshots with and without a preview in PPO is the cleanest way to settle video vs screenshots. On Google Play, a store listing experiment can test the promo video the same way.

Change one thing per test

If a variant changes captions and background color at once and wins, you do not know which change did it. Write a one-line hypothesis before you build anything, for example: "Leading with the core benefit in the first caption will increase first-time installers compared with leading with the app name." Then change only what the hypothesis is about.

One exception: with low traffic, a single bold test of a different concept can teach you more than many small ones, because small changes need large samples to detect. Once a concept wins, refine it one element at a time.

Producing variants is usually the slow part, especially across several localizations and device sizes. Appure lets you build and localize App Store and Google Play screenshots and exports every required size, so a caption or order variant takes minutes instead of a design cycle. It is free for one app. Check the current requirements on our screenshot sizes page before uploading.

Traffic, duration and avoiding false winners

Run full weeks

Store traffic and visitor behavior change by day of the week. Run every test for at least seven days, in whole weeks (7, 14, 21 days), so each variant sees the same mix of weekdays and weekends.

Do not stop at the first lead

Early results swing a lot, and a variant that leads for the first few days can end up flat or worse. If you stop the moment something looks good, you will often ship changes that do nothing. Decide the duration or the confidence level you need before the test starts, and stick to it.

Watch for seasonality and outside changes

• Avoid running tests over major holidays, sales events or your own launches, when visitors behave differently.
• Do not change other listing elements (icon, video, description) mid-test.
• Be careful when a large paid campaign starts or stops mid-test, since the traffic mix shifts.
• If your app is featured during a test, note the dates and consider re-running the test afterward.

Reading the results

Both consoles show a conversion difference with a measure of uncertainty around it. Read the range, not just the midpoint. A variant whose range runs from slightly negative to strongly positive has not proven anything yet. A variant whose whole range sits above zero is a winner, even if the estimated lift is modest.

If you tested several localizations, check each one. A caption that wins in English can lose in German because the translation runs longer and gets hard to read at thumbnail size.

A test with no winner is still useful: the change does not matter much to visitors, so you can stop debating it. A test your original wins saves you from shipping a change that would have cost installs.

When a variant wins, apply it, watch conversion for a few weeks to confirm the lift holds, and make it your new control. A sensible roadmap: settle the concept, then the first screenshot, then captions, then order and style. Mine reviews and support tickets for hypotheses, since the words users choose often make better captions than the ones your team writes.

When you do not have enough traffic

Small apps often find an experiment would take months to reach a result. Some ways around that:

• Test on the higher-traffic store first. Run tests where most of your visitors are and carry the lessons over. The audiences differ, so treat a transferred winner as a starting point, not a guarantee.
• Use fewer variants. One treatment against the original reaches a result fastest.
• Make bigger changes. A bold difference needs fewer visitors to detect, so set a realistic minimum detectable effect.
• Use ads and custom product pages. Send paid traffic to two custom product pages on iOS and compare them. Paid visitors may behave differently from organic ones, so treat the results as directional.

If you are not sure where your listing is weakest, run a free ASO audit first. It scores your title, description, screenshots, ratings and conversion signals and shows what to fix first, so limited traffic goes to the changes most likely to matter.

FAQ