Business
How to evaluate software demos before choosing a vendor

Stefan Weiss
To evaluate software demos, give each vendor the same important workflows, starting conditions and expected outcomes. During the session, record what you actually observe, what the vendor only describes and what remains untested. Compare those findings against agreed criteria before deciding whether to reject the option, ask a focused question or run a more representative test.
A demo can show how a product handles a particular scenario. It cannot establish every claim about integration, reliability, cost or use in your organisation. Your aim is to leave with enough comparable evidence to choose the next step.
This guide is for managers, product teams and consultants evaluating software for their own work. It covers preparation, useful questions, a fictional comparison and the review after the call. It is an editorial workflow, not a test of particular vendors or a complete procurement framework.
Before booking: define the decision and gather the inputs
Agree what the demos will help you decide. A first screening call might determine which products deserve closer investigation. A later session might examine a critical workflow before a trial. Those decisions need different depth.
Prepare four inputs: the problem you are trying to solve, the people affected, a small set of representative tasks and the constraints that could rule an option out. If these are disputed, resolve enough of the disagreement to write a coherent brief. A product tour will not settle an internal argument about whose work matters.
Identify the decision owner and invite people who understand the selected tasks. Include the person who handles an exception or maintains the result, not only the person who starts the workflow. Ask relevant technical or operational owners to define checks that the demo cannot resolve on its own.
Use requirements elicitation techniques when the underlying need is still unclear. Bring authorised synthetic or appropriately prepared examples where real material cannot be shared. Do not supply sensitive production data simply to make a demonstration feel realistic.
Expected result: a short brief that explains the next decision, the tasks to demonstrate and the people who can judge them.
1. Give vendors a common scenario brief
Describe a workflow from trigger to useful outcome. “Show us your reporting features” invites a tour of whatever looks strongest. “Show how an operations lead finds unresolved requests, checks their owners and produces a shareable update” gives the session something concrete to answer.
Include the starting conditions: the role using the product, the information already available and the state of the work. Add an ordinary exception, such as missing information or an absent approver, if that exception matters to the decision.
Ask the vendor to state any preparation, configuration, add-on or custom work needed for the scenario. If the demonstration uses a different plan or version from the proposed purchase, record that difference. Prepared examples can still be useful when their conditions are visible.
Montrium's vendor-selection guide recommends giving suppliers a script or focus areas and conducting a post-demo evaluation. It is vendor-authored guidance originally published in 2016, rather than independent evidence that one evaluation method produces better purchases. Montrium: conducting software demonstrations.

2. Agree criteria and evidence states before the first call
Choose criteria that reflect the selected work. Separate essential conditions from preferences. A polished optional feature should not cancel out a missing requirement that makes the workflow unusable.
For each criterion, decide what would count as adequate evidence at this stage. A demonstrated export may answer a simple format question. A proposed integration may need documentation and a later test in your environment. Use a separate evidence state so that an unanswered question is visible.
A practical set of states is:
Observed: the reviewer saw the stated behaviour under recorded demonstration conditions.
Claimed: the vendor said it was possible, but the session did not demonstrate it.
Not tested: the session did not address the criterion sufficiently.
Blocked or unsuitable: the evidence shows a specific obstacle for the proposed use.
These are editorial review labels, not a standard scoring system. “Observed” still has a boundary: it establishes what happened in that scenario, not performance in every setting.
If you use scores, write their meanings first and keep the evidence state alongside them. Do not quietly turn unknowns into zeroes or assume that all demonstrated behaviours are equally suitable. For help choosing priorities, see decision criteria.
3. Follow the work through to its outcome
During the session, ask the vendor to complete the selected task. Watch how information enters, how a person uses it, what happens at a handover and what comes out at the end.
Ask specific questions at the relevant point:
What has already been configured to make this work?
Which role can perform this action, and who can review or change it?
What happens if this required information is missing?
Can you show how the person corrects the result?
What can be exported, and what information is retained in that export?
Which part needs another product, paid capability or manual step?
What remains to be tested in our environment?
Let the vendor explain a legitimate limitation. An honest boundary can be more useful than a vague promise. If a different method achieves the required outcome, record the method and its implications rather than rejecting it solely because its interface differs from your current tool.
For AI features, include relevant difficult or ambiguous examples and inspect the output. Label Studio's guide to evaluating AI evaluation software recommends representative inputs and an agreed rubric; its recommendations concern that specific category and do not establish another vendor's quality. Label Studio: preparing useful software demos.
4. Separate the screen, the spoken claim and the follow-up
Keep a brief evidence note while the scenario is fresh. Record the criterion, what was observed, the conditions, unresolved questions and the next evidence needed. Where recording is permitted, note a useful time in the call as well.
An audio transcript can help establish what someone said. It cannot show that a visible workflow behaved as described. If a seller says “the export is complete”, the transcript preserves that claim; a reviewer still needs to inspect the export to confirm it.
Likewise, a screenshot of a completed result may not reveal the preparation or manual work that produced it. Keep the relevant context with the evidence. Use a permitted screen recording or reviewer notes when visual behaviour matters, following the permissions and information-handling rules for the session.
A public LinkedIn buyer account describes frustration with demos that translate a workflow problem directly into a feature pitch. Treat that as one person's experience, not a prevalence estimate. The practical response is to keep your scenario and unanswered questions visible throughout the call. Public buyer discussion.
5. Compare a worked example without hiding unknowns
Imagine a fictional team choosing software to manage internal equipment requests. It needs an employee to submit a request, an approver to handle it and an operations lead to identify work that has stalled. The following observations are invented examples, not results from tested products.
Criterion | Vendor A: fictional observation | Vendor B: fictional observation | Next evidence needed |
|---|---|---|---|
Submit an incomplete request | The interface prevented submission and identified the missing field | The seller described a validation rule but did not show it | Ask B to demonstrate the agreed missing-field case |
Handle an absent approver | The demonstrated workflow reassigned the request to another authorised reviewer | Reassignment was demonstrated after an administrator changed the setup | Confirm who can make the change and whether that dependency is acceptable |
Find stalled requests | A filtered view showed the sample overdue items | The prepared view did not include the required age information | Check B's proposed alternative against the same scenario |
Export the review record | A file was produced but its contents were not inspected | The seller promised a sample after the call | Inspect both files before treating the export requirement as confirmed |
The table does not establish a winner. It shows why a total score alone could mislead: several criteria still need evidence, and an administrative dependency may matter differently to different organisations.
Write a short conclusion about each essential condition. If B's missing export sample is the only unresolved issue, a focused follow-up may be enough. If neither product supports the necessary operating model, revisit the shortlist rather than rewarding the more polished presentation.
6. Debrief before memories become a consensus
Ask reviewers to record their own observations before discussing a shared conclusion. Then compare the evidence and resolve disagreements about what happened. Keep a difference in priorities separate from a disagreement about the observed behaviour.
A useful debrief asks: which essential conditions were addressed, which are still claims, which exceptions changed the assessment and what evidence could change the next decision? Refer to the relevant note, artefact or source passage when reviewers remember the same moment differently.
Record a proposed next step for each unresolved essential question, including an owner. Distinguish a vendor's commitment to send information from receipt and review of that information. Do not mark a criterion confirmed because the follow-up email has been promised.
Avoid completing an elaborate scorecard after the fact if the criteria were never agreed. First identify the decision you can support from the evidence you have. Document gaps and schedule only the work needed to resolve them.

Genuine Weeve interface with demonstration recordings. Select the relevant saved calls before comparing statements; the entries shown are synthetic examples.
7. Use Weeve to revisit the recorded discussions
When you have permission to record, Weeve can help retain the call and review spoken statements later. With Chat with recordings, select the relevant demo and follow-up discussions and ask a focused question across them.
For example: “For these selected vendor calls, list the spoken claims about request reassignment and export. Cite the supporting passages, identify promised follow-ups and keep conflicting or missing answers visible.” Check the source passages and whether the answer covers all the selected material.
The result helps you return to what was said. It is not an independent score of the software. Add your observations of the screen and inspect the actual files, documentation or trial results separately. A recording of a salesperson reporting a capability does not validate that capability.
Local recording, transcription and local AI workflows are available on your Mac with downloaded models. Choosing an optional external provider for chat changes where relevant text is processed. Check the selected mode and permissions before reviewing confidential vendor material.
Decide what comes after the demo
Choose the smallest next step that can resolve the remaining important uncertainty. That may be a written clarification, a representative trial, a technical proof of concept or a limited pilot. If a required condition clearly fails and no acceptable route is available, stop evaluating that option.
Keep commercial terms, security, integration and operational readiness with the people responsible for assessing them. A convincing demonstration can inform those checks without completing them. Record assumptions about plan, version, configuration and supporting work so they can be revisited if the proposal changes.
You are ready to move on when the decision owner can explain the evidence behind the next commitment, the conditions it depends on and the questions still open. That is a more useful result than simply agreeing which demo everybody liked best.



