How to choose the right AI solution
There’s a lot of pressure right now to do something with AI, but picking the wrong solution can be costly, distracting, and damaging. Here's the framework I’ve used over 20 years to separate hype from substance and evaluate solutions on the strengths of the benefits to the business.
According to Gartner: Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls, according to Gartner, Inc.
Choose the wrong system and you’re saddled with a service that doesn’t do what you want it to do, creates fractious calls with suppliers, unhappy users and ultimately fails to deliver the improvements to the business you were hoping for. I was chatting to a client earlier this week who was telling me that he was considering a vendor with a £1200 per user per month price tag, that can be a costly mistake.
Do nothing and risk being left behind by your competitors. Get it right, however, and the solution as a whole should become a good right hand man, freeing up your time to focus on activities that add more value and helping you, your team and your company to be more successful.
In this article I’m going to outline a framework that I use to structure how I choose and evaluate software. It’s something that I developed over 20 years ago with the help of Gartner and I have gone on to use it both as a purchaser and as a provider of software services. It’s robust and has helped me make fact based data driven software solution selection decisions rather than relying on an emotive reaction to just get something done.
Prerequisites
Any purchasing decision made by a business should be with the objective of helping the business to perform better and therefore the obvious place to start is to identify what you want a solution to help with, and yet I am constantly surprised by conversations that I have with leaders of enterprises who are under pressure to do something AI related, usually agentic, in the expectation that it will just magically make the business perform better. It doesn’t work like that.
As someone who has been creating AI products for the last 8 years and more general cloud software for a lot longer than that - let me say that there is nothing inherently different about an AI solution over any other software solution.
Identifying the problem to solve is the lynchpin that will enable the implementation of successful solution and measure the impact that it has on the business. By considering the most important problem in the business, whether it’s a failure to win enough bids, low csat/nps scores, or excessively high operational costs we constrain the problem we are trying to solve and create the conditions that will allow us to evaluate solutions. Action - Track and measure KPIs related to that problem and set a goal of what those KPI’s should be in order for that problem to be “solved” to the extent where it is no longer the biggest problem in the business.
Six Criteria
The system is based on six criteria (Product, Technology, Cost, Services, Vision, and Viability) and each is given a weighting to denote how important it is. Typically the weightings are apportioned like:
| Criteria | Weighting (%) |
|---|---|
| Functionality | 30 |
| Technology | 20 |
| Cost | 20 |
| Services | 15 |
| Viability | 10 |
| Vision | 5 |
I was initially surprised at some of the weightings, but discovered that this mix gave a balanced overview of the solution, the vendor and its services. The important thing is to ensure your team agree on the specific weightings prior to looking at any software.
Functionality
Now yes I know the marketing mantra “sell value not features”, but at the end of the day the functionality will determine what the solution is capable of doing, how well it does it and this will drive the value to solving the problem you are addressing.
I used to consider how the solution would support the business not just today but across a 3 to 5 year horizon. If the solution you are looking at has a long time to value like a data lake implementation that may still be about right, but with the pace of development of AI solutions and subscription based services that time horizon is too long.
I also try to think about the specific needs of the business, being a brit having an AI chatbot talk in US English might be a problem if I’m using its content externally. Does it support things like Multi-Factor Authentication or whatever your corporate standard is. Does it integrate with a range of external systems automatically like Glean or n8n or is it tied in to a specific vendor like Co-pilot is primarily to support the Microsoft system.
Does it work out of the box or will it need customisation or configuration effort to implement it properly. If it requires customisation is the product designed in a way that will allow you to do that internally (like Salesforce) or does it need to be done by the vendor ( like Clari).
Technology
I care about 2 things:
How reliable is the service going to be:
a. where can I access it from?
b. what happens in an incident / outage?
c. how are new releases and updates deployed?
d. how will it perform under the load you expect to use it.
e. SLAs on availability / uptime.
How secure is my data:
a. Does the solution exist in the corporate enterprise or is it delivered as a separate service?
b. Where is my data located?
c. How is my data protected whether at rest or in transit.
In the build versus buy debate the answers to these questions may be very different, and what’s right for your business will be depend on how critical the solution is (eg if it became unavailable could your business continue to operate), the nature of your business and sensitivity of the information it is processing (eg patient health records or LinkedIn marketing posts).
To help answer these questions I find resources like the Cloud Security Alliance Star Registry (CSA) to be useful which is a publicly available repository of reports made by vendors that describe their internal practices and procedures for things like Disaster Recovery, Business Continuity, Incident Management and Software Development. If the solution you are considering is not registered on a service like that it could be a red flag, if it is that could be starting point of a technical evaluation that your data security team could run.
Having been on both sides of the fence as a vendor and a purchaser, I have come to appreciate that it is completely normal for a security review to include extremely detailed descriptions of the solution architecture and provide evidence of things like penetration testing or disaster recovery plans.
Cost
I have always tried to establish the total cost of ownership, (ie in addition to the actual purchase price you also include support, implementation fees, even the time your organisation will incur. Using this in combination with the estimated time to value will give you an indication of the financial risk of any solution. The difficulty comes from comparing vendors with different pricing models. In the SaaS space the traditional per user licensing model is gradually being replaced by consumption based pricing models which suit the operation of AI businesses. Whilst consumption based pricing may sound nice in principle as you pay for what you use, the reality is that it makes it extremely hard to estimate in advance and this is an issue for both the vendor and the customer. Take Salesforce’s $2 per interaction for Agentforce led to some eye watering estimates and confusion as to what defined an interaction.
I find that the first step is to break out the one-off implementation costs from the ongoing recurring fees.
Implementation
One off fees usually come down to professional services for implementation (including training and configuration), the development of customisations and integration with other systems in the enterprise (eg a data lake).
Recurring
Recurring fees will cover subscription fees, annual support charges, upgrade fees, and on going training. You should also include things like internal MIS time if you need export data into an a data lake, or require specialist technical support if you are building the solution internally. Do not under estimate the time required to implement an internal system. These costs can and do mount up very quickly.
Services
The likelihood is that unless you have a very simple set of requirements you will need some help from the vendor to get your system up and running properly. The effectiveness of the implementation services and on-going support is essential to ensuring the long term success of the solution and I have seen projects completely fail in the past purely because of the quality of the people allocated by the vendor. Like costs I prefer to break out services into the one off implementation and the recurring support as they are usually two different groups within the vendor organisation.
Implementation
When it comes to implementation it is essential that there is a solid project delivery methodology that is appropriate for your organisation. It is normal for implementations to be divided into phases (sometimes referred to as sprints) and for each of these phases having a clear set of deliverables that will require your sign off at milestones. Make sure the vendor has a clear approach to implementation along with timescales that will give you confidence in their ability to deliver on time and in full. You should also establish what resources will be part of the project team and what their roles are (eg project manager, technical support, domain expert etc). Follow this up with calls to any references that the vendor is willing to make available for you to talk to.
Support
On going support can vary and are often best assessed through customer references, however the vendor should be able to provide you with details describing how their support service operates, when it is available, different levels of severity and response times.
Viability
With so many AI startups entering the market inevitably not all will survive and the reality is that many will disappear either through acquisition or just because they've run out of cash. I’m advocating that every vendor must be in the top right corner of a magic quadrant, but equally if the vendor you are considering has no track record of making a software product and no customer base then it’s probably too risk a proposition.
Financial Viability
I like to consider the risk of the vendor going out of business or being acquired and specifically what would happen to the service and your data if this were to happen. Responsible vendors should make provisions for these situations and be able to give you assurances for how your service will be handled.
Market Viability
How critical is the product to the overall success of the vendor? Is the offering a point solution from a vendor whose entire business is dependent on the success of that solution, or is it a module from a tech giant who may decide to drop the whole product line if they don’t get traction with sales.
Organisational Risk
Understanding the strength of the organisation and its commitment to long term business development is important for any supplier. Has the product been designed from the ground up to support large numbers of clients, does it depend on a single LLM provider. What is the strength of the vendors partner network and their ability to pool resources and expertise where required?
Vision
Often overlook are the long term plans of the vendor, its products and services. Gartner recommend that you divide this into 3 components (product, service and corporate), but generally I have found it adequate to consider a few questions: The first aspect to consider is whether the general direction of the vendor and the product roadmap is aligned to your own long term requirements. For example if you are looking to create a solution to make bid writing more effective then considering a vendor who is committed to developing capabilities for lead generation may not be a sensible idea.
Conclusion
Make sure that you are able to follow up on customer references over the phone and are armed with specific questions, that you can ask, bearing in mind that references are going to be positive about the vendor. Ask about the stages of implementation, how long it took, what support issues have arisen and how long it’s taken to get them resolved.
Above all try to keep in mind that AI isn’t magic and until the day arrives when we have AGI to take over all of our roles, businesses will thrive on the quality of people and the quality of the tools and knowledge that they have available. Making a decision about an AI solution should always be driven by the needs of the business and using a structured framework like the one outlined here can help maximising the chances of it giving you the benefits you hope.
