The release of ChatGPT in November 2022 sparked excitement about the possibilities of Generative AI. In just five days, it gained over a million users. As businesses began to explore how to leverage this technology, they quickly realized that using AI internally was crucial.
Fast-forward three years, and many companies have jumped on the Generative AI bandwagon, building their own GPT or LLM-based solutions. However, what I often see is that creators simply add an LLM with Retrieval Augmented Generation (RAG) to their data, hoping for the best. But does this approach truly deliver value? I have some doubts.
This sentiment is also confirmed by various studies, MITs “State of AI in Business 2025” reports that “Despite $30–40 billion in enterprise investment into GenA (…) 95% of organizations are getting zero return”. In this post we speak about software companies or companies that have at least their own IT department and are planning to build their own AI solutions.
The naive approach for Generative AI projects
Cost pressure is high in many tech companies. And so is the hype around AI. An easy strategy is therefore to just invest in AI. An LLMs capability can look impressive, but the main issues is that it doesn’t have access to company data out of the box. Retrieval augmented generation can help there and extend the LLMs context by data in files or on wiki pages. It’s also not super difficult to implement. So, many just start to RAGify everything they have. This is nice to have and the use cases around it are many, but:
- The data that is retrieved is often in bad quality and can be outdated.
- As the use cases are endless, end-users struggle to understand how to exactly use the new possibilities. I’ve got that wiki page no in a chat bot, nice, but what now? What should I do with it?
- Even if there is a chat bot available somewhere backed by RAG, it’s usually a standalone app. This requires users to switch context and leads to a fragmented tool landscape.
- The return on investment (ROI) of such solutions is not measured or not understood.
- Projects look fancy in the beginning because you get impressive answers very soon. But optimising the solution to reach production state is hard. Project teams often miss to have an evaluation pipeline and defined KPIs.
- Big use cases are tackled early on and take months to implement, delaying the benefits and in the worst case leading to a solution that is not usable at all.
- The core of the AI solution might be built within a few days. But integrating it into the existing system landscape, considering security, compliance, people training, data maintenance, and other factors can take months.
Still the pressure remain, we have to do something with AI! And in the next section I would like to highlight a few points that help you to make your project a success.
How to tackle Generative AI projects
I’m convinced that AI has benefits in it for most businesses. How do we make it a success? First of all, slow down! Don’t follow the latest trends daily, but have a robust AI strategy that is dynamic enough to adapt, but doesn’t lead to jumping always onto the latest technological advances in the field of AI. Because there’s just too much going on.
And then more importantly. Don’t take Generative AI and RAG as a technology and ask yourself what it can solve. That’s the wrong direction! Ask what problems do you have in your company and design a solution for it. And then ask what parts of the solution can be covered or improved by generative AI.
Also don’t forget how software was built in the last years. Don’t forget agile! Put the user, feedback and short iterations into the center.
Consider the following when you’re planning to build an AI solution in your organisation:
- Start with problems, not technology: Design a solution for specific company challenges, and then ask how AI can help.
- Use agile principles as in any other software product you have built in the last years.
- Don’t build standalone solutions, but make sure AI is integrated into the existing workflows.
- Educate and train the user. This starts even by making sure the AI strategy is understood by everyone and nobody has to fear for their jobs.
- As AI is non-deterministic, define KPIs that you need to achieve to make your solution useful. Iterate and improve.
- Define the ROI you want to see when you plan the solution. What is your business case? What can your colleagues do with the solution you’re building?
- Consider data management, security, compliance, and other non-functional requirements early!
Finally, you might put a generative AI solution into production that makes some work faster or better. Ask yourself what the time freed up by that, can be used for. If you make a process faster only to run into waiting times or have colleagues who can’t use their newly won time, you’ve failed too.