Nine Questions to Ask Before Choosing AI for Your E-commerce Store

A shopper arrives on a product page. The product is close to what they want, but something is wrong: the size, the style, or the price. Your catalog may hold a better match.

What can your store learn from that visit, and can it use that information while the shopper is still there?

That is a useful place to start when evaluating AI. It gives you something concrete to investigate: what the system notices, what it does next, and whether that response helps the shopper find something they want.

You may already have tried a recommendations app, worked with a vendor that struggled with your catalog, or spent months building something internally. It is reasonable to want a clearer explanation of why the next attempt should work.

An AI-powered product solution, as we are using the term here, is software that helps decide which products and content a shopper sees. It might shape search results, recommendations, category pages, or a discovery feed on a product page. Those are different places to intervene. The right one depends on where your shoppers need help.

Here are nine questions to ask any vendor, including us. There is also a scorecard at the end to help your team compare the answers.

Start with the visit you want to improve

Before taking a demo, choose a problem specific enough to investigate.

Suppose a new visitor arrives directly from an ad, a search result, or an AI assistant. That product page is their first experience of your store. If the product is wrong for them, what helps them discover something better?

Look at three things:

• Where do new visitors enter, and what can they discover from that page?
• What happens when the first product is not a good match?
• Which relevant parts of your catalog are difficult for these shoppers to find?

Keep a few actual products and customer journeys in mind. They will make the vendor's answers much easier to evaluate.

1. How was the lift measured?

To understand a lift percentage, you need to understand how it was measured.

The underlying question is simple: what would have happened to comparable shoppers without the tool?

Ask the vendor:

• What was the comparison? Were shoppers randomly assigned to the proposed experience and the existing site during the same period? Comparing this month with last month leaves room for promotions, seasonality, and changes in advertising to explain the difference.
• Lift in what? Revenue per visitor or session, conversion rate, and average order value answer different questions. Ask for the metric definition, the number of shoppers, and the uncertainty around the estimate. For revenue per visitor or session, the calculation should include people who did not buy.
• Does the result cover the full test? Ask how the overall result was calculated and whether weak days were included alongside strong ones.
• What was the stopping rule? A fixed test plan can work, as can a statistical method designed for decisions as data arrives. Stopping at a favorable reading requires the right method; otherwise it can make the evidence look stronger than it is. [1]
• What does the average represent? Ask how results were combined across stores, how many tests were included, and how widely the outcomes varied. Results from stores like yours are more informative than an unexplained “up to” figure.

Look for: a test design your team can inspect, a clearly defined outcome, and an honest account of uncertainty.

A reason to pause: the vendor keeps returning to the percentage when you ask how it was produced.

2. What can it do for someone who has never visited?

A salesperson can begin helping someone without knowing their purchase history. They can notice what the person looks at, listen to a question, and adjust their next suggestion.

An online system has different signals. A click, a pause, or a skipped product may provide a clue, but none tells you with certainty what a shopper wants. The useful question is how the system responds to that limited information.

Ask the vendor:

• What does a new visitor see before taking any action?
• Which actions can change the experience during that same visit, and how quickly?
• Does it need a login, a previous purchase, or third-party data to be useful?
• What information does it use when it knows very little about this shopper?

Try a visit yourself in a private browser window. Arrive through a product link, browse, and ask the vendor to explain the decisions you see.

A changing recommendation does not, by itself, demonstrate continuous model learning. A system can use fresh input without retraining its model. Ask what actually updates, and use a controlled test to assess the benefit. [2]

Look for: a useful first visit, with a clear explanation of how new information changes the next decision.

A reason to pause: the proposed benefit depends on shoppers returning, even though your problem is helping them on their first visit.

3. What happens when shopper interest changes?

There are two changes to think about. One shopper can change direction while browsing. Patterns across shoppers can also change when a campaign launches, a product becomes popular, or the mix of incoming traffic shifts.

A system needs an answer to both.

We encountered this while building Product Genius. At Betsey Johnson, revenue lift could remain steady for a week and then disappear within hours, even though we had made no change. By the time we could confidently detect a shift in the aggregate results, our response could already be late. We describe that experience in our technical whitepaper. [3]

That experience made the timing of adaptation central to our design. It also clarified two separate jobs: deciding what to show a shopper now, and measuring whether those decisions improve outcomes over the full experiment.

Ask the vendor:

• How quickly can the system respond to an individual shopper?
• How does it learn when patterns across shoppers change?
• What happens while it is waiting for enough new data to update?
• Can you show results through changing conditions, including periods when performance weakened?

Look for: a concrete explanation of what updates, when it updates, and evidence of how the system behaves through changes.

A reason to pause: adaptation means someone on your team must notice the change and rewrite the rules.

4. Can it handle the difficult parts of your catalog?

Your hardest category is a useful place to test the system. That might mean products with many sizes, one-of-a-kind inventory, or items whose appeal is difficult to capture in a short description.

The shopping context matters, too. Someone who has just bought a wallet may have finished that particular task. Someone buying a t-shirt may want another color. Those are possibilities for the system to investigate, with the shopper's behavior providing more evidence.

Ask the vendor:

• Can we test our hardest category, using our actual products?
• How does it handle sizes, variants, and an item that sells out?
• How soon can a new product appear, even if it has no sales history?
• How does a purchase change what the shopper sees next?
• What has been difficult in a catalog like ours, and how did you address it?

Look for: a demonstration that exposes the difficult cases and explains how they are handled.

A reason to pause: “it works for any catalog” is the whole answer.

5. Can your team inspect and direct its decisions?

If the system changes what shoppers see, your team needs a way to question those choices. That becomes especially useful when a result is surprising.

A readable explanation helps, provided it connects to what the system actually observed and did. A convincing story about a decision is not enough to establish why it happened or how much revenue it caused.

Ask the vendor:

• Can we see the information behind a recommendation and the decision that followed?
• Can our merchandising team understand and challenge the patterns it reports?
• Can we give it a seasonal priority, exclude a product, or set a constraint, then verify that it followed the instruction?
• When a dashboard assigns revenue to a pattern or recommendation, what does that number measure?

That last question matters. Revenue associated with a recommendation and revenue caused by that recommendation are different claims. A test of the whole system does not automatically establish the contribution of every individual decision.

Look for: actual decisions your team can inspect, useful controls, and clear boundaries around what the reporting can establish.

A reason to pause: explanations stay vague, or your team cannot correct behavior that conflicts with how you run the business.

6. What happened to profit?

Ask what it cost to produce the additional revenue.

Discounting may bring in additional orders while reducing the margin on each one. The business question is whether the additional sales leave you better off after product costs, discounts, returns, and fees.

The same care applies to discovery. A shopper looking for something unusual may find a baseball-themed purse much more appealing than a standard black bag. Helping that shopper find it could create value. Whether it improves your economics is something to measure.

Ask the vendor:

• How much of the change came from full-price products versus discounted products?
• What happened to average order value, returns, and profit contribution per visitor?
• Did shoppers discover and buy relevant products beyond the usual bestsellers?
• What cost data would you need from us to assess profitability?

This standard applies to our own results. The tests described in our technical whitepaper measured revenue per session; the system did not have product-level gross margin data. Revenue lift alone does not establish that profit increased. [3]

Look for: a clear account of the sales mix and a practical way to assess profit using your cost data.

A reason to pause: the vendor treats a revenue increase as sufficient proof that profit increased.

7. What work will your team take on?

The people running your store already have work to do. Installation, catalog cleanup, rule maintenance, and troubleshooting all belong in the evaluation.

Ask the vendor to walk through an ordinary week after launch. Then ask about a week when something goes wrong.

• Who handles installation, and how many hours will our team contribute?
• What recurring work remains with merchandising, marketing, and engineering?
• Who maintains rules, handles exceptions, and investigates a decline in performance?
• How will we check mobile page performance, and what happens if the service is slow or unavailable?
• Who runs the test, what does it cost, and what happens if we decide to stop?

Look for: named owners, realistic estimates of your team's time, a test plan, and a clear support process.

A reason to pause: “it's easy” never turns into an explanation of who does what.

8. What is live today?

Ask the vendor to separate capabilities you can test now from beta features and planned work. If a feature matters to your decision, ask to see it working.

9. Which parts does the vendor control?

Using outside models or services can be a reasonable engineering choice. You still need to know who can fix a problem, what depends on another provider, and how those dependencies affect cost and reliability.

Want to understand the full methodology? Request a demo and bring us your hardest category.