Before You Sign That AI SaaS Contract: Seven Questions Worth an Awkward Silence

I have sat through a lot of AI software demos over the last two years. They almost always follow the same script. A clean example document goes in, a perfect answer comes out, and someone in the room says "that would save us so much time." Nobody asks about the other four hundred documents that look nothing like the example.

That is not really a criticism of vendors. A demo is supposed to show the best case. The trouble starts when an SME buys on the best case and meets the average case three months after go-live, when the annual invoice has already been paid.

So here is the list I wish more buyers brought to those meetings. None of these questions are hostile. A good vendor will have answers ready. A weaker one will suddenly become very keen to schedule a follow-up.

1. What is the accuracy on our data, not yours?

Ask for a pilot on a sample of your real documents, emails or orders. Not fifty hand-picked ones. Two hundred, taken at random from last month, including the messy ones with handwritten notes and forwarded threads. Then ask for the result per field instead of one overall percentage. "92% accurate" can easily hide a delivery date field that is wrong one time in four, and that happens to be the field your planners care about most.

2. What does the tool do when it is not sure?

If I could only ask one question, it would be this one. Every model is uncertain some of the time. The difference between a useful tool and a risky one is whether that uncertainty is visible. A system that flags a low-confidence value for review is doing its job. A system that fills in a plausible guess, with the same confidence as everything else, is quietly moving risk into your process.

3. Which model sits underneath, and what happens when it changes?

A lot of AI SaaS is a well-designed interface on top of a model from one of the large providers. That is perfectly fine. But models get updated, retired and repriced, and a prompt that worked well on one version can behave differently on the next. Ask how the vendor tests for that, and whether you hear about a model change before it happens or after.

4. Where does our data go?

• Which country is it processed and stored in? For most Dutch companies, "within the EU" is the minimum answer you want to hear.
• Is it used to train anything? Get the answer in the contract, not on a FAQ page.
• Which subprocessors touch it, and can you see the list?
• How long is it kept after you cancel?

5. How is it priced when usage doubles?

Per seat, per document, per token, per "credit". Every model is fine until your volume changes. Run the numbers for your busiest month rather than your average one, and for the version of your company that is thirty percent bigger in two years. Per-document pricing that looks cheap in a pilot has a way of becoming the largest software line in the budget once the whole inbox runs through it.

6. Can a person see why it did what it did?

When a value is wrong, somebody on your team will have to figure out why. Can they see which sentence in the source the value came from? Can they see which rule fired? If the answer is "the model decided", your people will end up re-checking everything by hand, and the time savings quietly disappear.

7. How do we leave?

Nobody enjoys this question in a sales meeting, which is exactly what makes it useful. Can you export your data, your corrections and your configuration in a format another system can read? The corrections matter most. Every time your team fixes a wrong value, they are building a dataset that is specific to your business. That dataset should belong to you.

The part nobody puts in the demo

The licence is usually the smallest part of the cost. The bigger parts are connecting the tool to your ERP or mailbox, tidying the master data it depends on, and the time your people spend reviewing the cases it flags. None of that fits in a twenty-minute demo. All of it decides whether the tool is still being used a year from now.

None of this means you should avoid AI SaaS. Buying is often exactly the right call. Just buy it the way you would buy a machine for the shop floor: tested on your own material, with a clear idea of what happens on the day it breaks.