In the last eighteen months, I have had more conversations about AI tools than about any other topic. Most of these conversations follow the same pattern. A business leader has seen something impressive: a demo, a competitor’s announcement, or a LinkedIn post. They want to know if they should be doing the same.
The honest answer is almost always, “It depends on a question you haven’t asked yet.”
The problem with AI tool pitches
AI products tend to perform well in demonstrations. They have a well-chosen example, a clean interface, and a confident presenter. However, the gap between how a tool appears in a ten-minute demonstration and its actual daily performance is often significant.
This isn’t unique to AI. However, AI tools have a particular version of this problem: they excel on easy cases and quietly fail on hard ones. For example, a document summarization tool that produces a flawless summary of a clean, well-structured PDF may perform poorly with messy, scanned, and inconsistently formatted documents, which are what your team actually works with. You won’t find that out from the demo.
Another issue is that most AI tools are sold based on capability rather than fit. “It can do X, Y, and Z.” The relevant questions are whether your business needs X, Y, or Z, how often it needs them, and whether the current method is painful enough to justify the cost and change.
Start with the problem, not the tool
Before evaluating any AI product, it helps to write down what problem you are trying to solve in plain terms.
Not, “We want to use AI.” Be more specific. For example, “Our team spends three hours a week summarizing customer feedback into reports.” “We waste time whenever a new employee tries to find information that is scattered throughout our documentation.” “Our sales team writes the same five types of emails repeatedly.”
If you can’t write that sentence, the tool evaluation is premature. You’re shopping for a solution before confirming that you have a problem.
Once you clearly state the problem, the evaluation becomes much simpler. Does this tool reliably solve that specific problem with the kind of inputs you actually have? Anything else is a feature you’re paying for but not using.
What to actually test
A proper evaluation of an AI tool should use your actual data, workflows, and edge cases rather than the examples provided by the vendor.
Ask for a trial period. Most legitimate tools offer one. During the trial, test the tool on the ten tasks from the past month that the vendor claims it can handle. Not hypothetical tasks. Use the actual documents, data, and inputs involved.
Pay attention to the failure cases. Every tool fails sometimes. What matters is how it fails. Does it produce a confidently wrong answer? Does it silently skip difficult sections? Does it handle your specific document format, industry terminology, and language? A tool that admits when it’s unsure is more useful than one that makes something up without indicating as much.
Also, test what happens when the inputs are messy. They usually are.
The questions worth asking before you sign anything
- Exactly what does this replace? If the tool doesn’t replace or reduce a specific task or cost, then it’s an addition to your workload, not a reduction.
- Who on the team will use this tool, and how often? A tool that requires a twenty-minute setup each time it’s used or only works well with a specific type of prompt will stop being used within a month. Adoption is part of the evaluation process.
- What happens to your data? This matters more than most vendors acknowledge in their pitches. Where is your data processed? Is it used to train the model? Is it stored, and if so, for how long? For businesses that handle client data, financial records, or anything else sensitive, these are not just preference questions, but compliance questions.
- What is the cost at actual usage levels? Many AI tools use usage-based pricing. The trial might be free or inexpensive. Get an estimate of what your monthly bill will look like at realistic usage levels for your team. Then, compare that to the time or cost you’re actually saving.
- What’s the exit path? If you build a workflow around this tool and the pricing changes or the company goes out of business, what will you do? Is your data exportable? Can the process be recreated without the tool?
The tools worth being skeptical of
Any tool that is essentially a wrapper around a general-purpose AI API with an attractive interface and a price tag specific to a particular industry deserves extra scrutiny. These can be perfectly good products. However, you should know that you’re mostly paying for the interface and prompt engineering, not a unique underlying capability. Sometimes that’s worth it. Other times, the underlying API is available directly at a fraction of the cost if someone on your team spends an hour setting it up.
Be skeptical of annual contracts pushed during the first conversation. A vendor who won’t let you run a proper trial before committing has more confidence in the demo than the actual product.
Be wary of tools that promise to “transform” a process without explaining specifically how they will do so. “Transformation” is a marketing word. What you want to know is which step gets faster, by how much, and what the output will look like.
A practical filter
Before purchasing any AI tool, I suggest answering these five questions:
- What specific task will this replace or speed up? How much time does that task currently take?
- Have we tested it on our actual data, including messy cases?
- What happens to our data?
- Does the cost make sense for our expected usage?
- Is there someone on the team who will be responsible for this tool and ensure that it continues to work properly?
If any of these questions don’t have clear answers, the decision can wait. The tools will still be there next month and will probably be cheaper.