Many companies I see treat implementing AI as a large initiative. They plan a multi-year program and buy tools and platforms, but in many cases there is no measurable performance. Did AI work? How do we measure the result? What problem were we trying to solve? I think the real problem is that there is more emphasis on using AI than on how AI can actually help. The goal should be to provide staff with an armada of tools, start with a minimum viable plan, deploy and monitor, assess and adjust, then repeat as we collect more feedback. This doesn't mean giving everyone ChatGPT. It means matching a specific tool to a specific problem.
Treat AI like any other system
I think we should look at implementing AI in an organization the same way we look at any other system. Stop the waterfall approach and use agile. Start with the problem, and how AI can help people do their job better. Each company is different, so there is no single answer, but the question is the same everywhere.
I call the big "transform everything" program "Big T." These programs stall because people resist when whole departments change at once, and because AI models change every few months, so a 3-year plan is out of date before it ships. The alternative is "small t": targeted fixes, small pilots, and value in weeks instead of years. In the Brynjolfsson study, customer support agents with an AI assistant were 14% more productive on average.
Here is what that armada of AI tools can look like in a real company. AI can work as a clerk that does admin work like summarizing tickets, as a colleague that writes first drafts a person reviews, or as a coach that looks up a policy while the customer is still on the call. Each one is a specific tool for a specific job, and every team should have access to the ones that fit their work.
Start simple
On the technical side, start with the simplest option that works. Better prompts may be enough. If the model needs your internal documents, use RAG (feeding your own documents to the model when someone asks a question). Fine-tuning makes sense for some problems, but it shouldn't be the default. Embedding a million pages for RAG costs about $16 for the embeddings alone. Storage, integration, and maintenance add more, but cost is rarely the main problem. Old systems and messy data usually are.
A plan for one quarter can be simple. Pick a few workflows that are repetitive and costly, prepare the documents behind them in a test environment, then ship in short sprints and measure accuracy and how many people actually use it.
Overall, I think we should treat AI like any other system we build, and stop waiting for the big plan.
The slides are below.
Note: Much of the content in this article is taken directly from or inspired by the work of Andrew McAfee of the MIT Sloan School of Management. Related reading: Kevin J. Delaney, "The low-hanging gains from AI for businesses and how managers can use AI as clerk, coach, and colleague," Charter, April 21, 2024 (an interview with Andrew McAfee); and Kristin Burnham, "How to kick-start your generative AI strategy," CIO, January 25, 2024.