Enterprise AI requires more than the capabilities of general-purpose Large Language Models (LLMs). Retrieval-Augmented Generation (RAG) bridges this gap by grounding AI responses in an organization’s trusted data through a pipeline of chunking, embeddings, vector databases, and context-aware retrieval. This overview examines the key challenges organizations must address—including hallucinations, security and access control, data quality, and knowledge freshness—while highlighting the program manager’s critical role in governing, scaling, and evolving RAG from an initial pilot into a secure, reliable, enterprise-grade platform.
Generative AI is impressive when it can answer general questions. But for enterprises, the real value begins when AI can securely access, understand, and use the organization's own knowledge. Consider the information generated inside a large organization every day: policies, procedures, contracts, project plans, technical documentation, product specifications, regulatory requirements, incident records, lessons learned, customer-support information, and years of institutional knowledge.
A general-purpose Large Language Model (LLM) may understand project management, finance, healthcare, technology, or operations—but it does not automatically know what is contained in those internal systems.
This creates one of the fundamental challenges of enterprise AI: How do we allow an AI system to use trusted organizational knowledge without retraining the model every time that knowledge changes? One of the most important architectural approaches for solving this problem is Retrieval-Augmented Generation, or RAG.
RAG allows an organization to connect an LLM to trusted enterprise information so that responses can be generated using relevant organizational knowledge rather than relying solely on what the model learned during training. But RAG is often oversimplified. You may hear it described as "connecting company documents to an LLM." Technically, that captures part of the idea, but enterprise RAG is much more than a document connection. It is an architecture involving data pipelines, embeddings, vector databases, retrieval mechanisms, security controls, governance, monitoring, and operational processes.
What this means for Program Managers and Business Leaders
For program managers and business leaders responsible for AI transformation, understanding this architecture is becoming increasingly important.
You do not need to become a data scientist to understand RAG. But you should understand how the major components work together, where failures can occur, and what must be governed as the technology moves from experimentation into production. At a high level, the architecture looks like this:
Enterprise Data → Chunking → Embeddings → Vector Database → Retrieval → Prompt → LLM → Grounded Response
Let's look inside the architecture.
Step 1: Enterprise Data Provides the Knowledge Foundation
Every RAG system begins with data.
Before organizations start debating which LLM, vector database, or AI platform to use, they need to determine what knowledge the AI system will be allowed to access.
That knowledge could include:
- Policies and procedures
- Project documentation
- Technical documentation
- Contracts
- Product manuals
- Knowledge-base articles
- SharePoint content
- Customer-support information
- Regulatory documentation
- Lessons learned
- Standard operating procedures
- Training materials
- Architecture documentation
- Incident and problem-management records
This sounds straightforward until you consider the complexity of enterprise information.
- Which version of a policy is authoritative?
- Who owns the document?
- When was it last updated?
- Who should have access to it?
- Does it contain confidential, regulated, proprietary, or personally identifiable information?
- Should every employee receive the same answer?
The first architectural challenge in RAG therefore isn't artificial intelligence. It is information governance. Organizations need processes for determining which sources are trusted, who owns them, how they are classified, how long they are retained, and which users or systems should be allowed to retrieve them. If the underlying knowledge base is poorly governed, the AI system can simply make that problem more visible—and at much greater scale.
A useful principle for enterprise leaders is: RAG cannot create trustworthy organizational knowledge from untrustworthy source information. The quality of the AI experience begins with the quality and governance of the enterprise data behind it.
Step 2: Documents Are Broken into Chunks
An enterprise may have thousands—or millions—of pages of information. Sending every relevant document to an LLM each time someone asks a question would be inefficient, expensive, and frequently impossible because models have limits on how much information they can process within a single interaction. Instead, documents are divided into smaller sections known as chunks. Imagine a 60-page project governance manual. Rather than treating the entire manual as one massive piece of information, the RAG pipeline might divide it into smaller passages covering topics such as:
- Risk Management
- Change Control
- Financial Governance
- Escalation Procedures
- Steering Committee Responsibilities
- Project Closure
When someone asks about escalation procedures, the system does not need to send all 60 pages to the model. It can retrieve the portions most relevant to that question. However, chunking introduces an important design decision. If chunks are too large, they may contain significant amounts of irrelevant information. If they are too small, important context may be separated from the information needed to interpret it correctly.
For example, a paragraph describing an approval threshold might be meaningless if the heading identifying which business unit the threshold applies to was placed in a different chunk. This is why chunking is not simply a technical preprocessing task. It directly affects retrieval quality and ultimately the quality of the AI's answers.
Step 3: Embeddings Turn Meaning into Numbers
Once documents have been divided into chunks, the system needs a way to determine which chunks are conceptually related to a user's question. That is where embeddings become important. An embedding is a numerical representation of meaning. Rather than representing only the literal words in a sentence, embeddings capture relationships between concepts in a multidimensional mathematical space.
Consider these two phrases: "Project risk management" and "Identifying and mitigating delivery threats."
The phrases contain different words, but they describe closely related concepts. Traditional keyword search may struggle to recognize that relationship because the exact terminology differs. Embeddings make it possible for the system to recognize semantic similarity. This is one of the reasons semantic search can be significantly more powerful than simple keyword matching in enterprise knowledge systems. The same concept applies to the user's question.
When someone submits a question, that question can also be converted into an embedding. The system can then compare the mathematical representation of the question with the mathematical representations of the organization's document chunks. This allows the system to ask: Which pieces of enterprise knowledge are closest in meaning to what this person is asking?
Step 4: The Embeddings Are Stored in a Vector Database
Once embeddings have been generated, they need to be stored somewhere where they can be searched efficiently. That is the role of a vector database. Traditional databases are extremely good at answering questions based on structured or exact information. For example:
- Find customer ID 10452.
- Show all projects with status = Red.
- Return invoices created after August 1.
Vector databases are designed to support a different type of search. Instead of asking: "Which record exactly matches this value?" the system can ask: "Which pieces of information are most similar in meaning to this question?"
When the user's question is converted into an embedding, the system compares that vector with vectors stored in the database and identifies the chunks that are semantically closest to it. This is what enables the system to retrieve relevant information even when the user does not use exactly the same terminology as the source documents.
Step 5: Retrieval Finds the Most Relevant Information
Now imagine that a program manager asks: "What is our escalation process for a critical production issue?" In a traditional chatbot, that question might be sent directly to the LLM. In a RAG architecture, something important happens first. The system searches the approved enterprise knowledge base. It may retrieve information covering:
- Severity classifications
- Incident escalation procedures
- Required stakeholders
- Response-time expectations
- Communication requirements
- Executive-notification requirements
The system then selects the information most relevant to the question. This is the retrieval part of Retrieval-Augmented Generation. And retrieval quality is one of the most important factors determining whether a RAG application succeeds.
A sophisticated LLM cannot compensate for consistently retrieving the wrong information. This creates a useful way to think about RAG failures. When the AI provides a poor answer, the problem may not necessarily be the LLM. The system may have:
- Retrieved the wrong document.
- Retrieved the wrong section of the right document.
- Failed to retrieve an important section.
- Retrieved an outdated version.
- Retrieved information without enough surrounding context.
That distinction matters because organizations need to diagnose the entire pipeline rather than assuming every incorrect answer is an "LLM problem."
Step 6: The Retrieved Information Becomes Context
Once the relevant information has been retrieved, it is added to the prompt supplied to the LLM. Conceptually, the instruction might look something like: Using the following approved company documentation, answer the user's question. If the information required to answer the question is not contained in these sources, indicate that you do not have enough information.
The retrieved passages are then supplied as context. This fundamentally changes the model's task. Instead of asking: "What do you know about this subject?" we are effectively asking: "Based on this trusted information, what is the answer?"
That distinction is at the heart of enterprise RAG. The model's general reasoning and language capabilities are combined with organization-specific information retrieved at the time of the request. This also means organizations can update the knowledge available to the AI without necessarily retraining the underlying LLM every time a policy, procedure, product specification, or business rule changes.
Step 7: The LLM Generates a Grounded Response
The LLM now receives two important things: The user's question and The enterprise information retrieved to answer it. The model uses that context to generate its response. For example: "Critical production incidents must be escalated to the Incident Manager and application owner within 15 minutes. Executive notification begins for Severity 1 incidents following the initial assessment."
A mature enterprise RAG application may also provide citations or links to the source material supporting that response. That capability is extremely important. Users should not simply be expected to trust an AI-generated answer because it sounds authoritative. They should be able to ask: Where did this information come from? This creates something enterprises need from AI systems: Traceability.
For regulated industries, audit environments, legal workflows, healthcare, financial services, cybersecurity, and other high-risk use cases, traceability can become a fundamental requirement rather than an optional feature.
The RAG Architecture at a Glance
Each stage in the pipeline plays a distinct role in determining the quality, accuracy, and trustworthiness of the final answer. A weakness at any point — from source data governance to retrieval ranking — can degrade the response the user receives. Understanding this end-to-end flow is essential for program leaders responsible for RAG delivery.
Why RAG Can Reduce Hallucinations—but Doesn't Eliminate Them
One of the biggest misconceptions surrounding RAG is that implementing it eliminates hallucinations. It does not. RAG can substantially improve grounding because the model receives relevant enterprise information before generating its response. But multiple failure points still exist.
- The system could retrieve the wrong document.
- The source information could be outdated.
- Important context could have been separated during chunking.
- Permissions could be incorrectly configured.
- Retrieval rankings could prioritize the wrong information.
- The LLM could misunderstand or incorrectly summarize the retrieved material.
- The response could contain a citation that does not actually support the statement being made.
For these reasons, organizations need to evaluate the entire RAG pipeline, not simply the performance of the LLM. This is a critical distinction for leaders evaluating AI solutions. The quality of a RAG application is determined by the interaction between multiple components.
Source Quality × Chunking Quality × Embedding Quality × Retrieval Quality × Prompt Quality × Model Performance × Governance
Weakness in any one of those areas can affect the final response.
Security and Access Control Cannot Be an Afterthought
Enterprise RAG introduces another challenge that pilot projects can easily underestimate: authorization. Suppose an organization indexes HR policies, financial information, legal documents, customer contracts, engineering documentation, and executive strategy documents. A technically successful retrieval system may be capable of finding all of them. That does not mean every employee should be able to retrieve all of them.
The AI experience therefore needs to respect the organization's existing security model. A user asking a question should only receive information that person is authorized to access. This becomes increasingly complex as organizations connect multiple repositories and data sources. Program leaders should be asking questions such as:
- Are permissions applied during retrieval?
- What happens when an employee changes roles?
- How quickly are access changes reflected?
- Can sensitive information accidentally appear in retrieved context?
- Are prompts and responses logged?
- How long are those logs retained?
- Can administrators audit which sources were used to generate a response?
Enterprise AI governance must therefore extend beyond the model itself to the data and retrieval layers surrounding it.
Freshness Is Another Hidden RAG Challenge
Enterprise knowledge changes constantly.
- Policies are revised.
- Contracts are amended.
- Technical documentation changes after releases.
- Product specifications evolve.
- Projects close.
- New lessons are learned.
- Employees change roles.
If the RAG knowledge base is not updated, the AI can confidently retrieve information that was correct six months ago but is incorrect today. Organizations therefore need a strategy for knowledge freshness. When a source document changes:
- When is it reprocessed?
- When are its chunks regenerated?
- When are embeddings refreshed?
- How is the old information removed or superseded?
- How does the system determine which version is authoritative?
These may sound like operational details, but at enterprise scale they become governance and service-management concerns. A RAG system is not something organizations simply deploy and forget. It requires an operating model.
RAG Is a System, Not a Feature
This may be the most important concept for program managers. An enterprise RAG solution is not simply: "Connect our documents to ChatGPT." A production RAG environment is an ecosystem involving:
Data → Security → Cloud Infrastructure → AI/ML → Application Development → Compliance → Business Operations
The original architecture therefore expands significantly when viewed through a program-management lens. Program leaders may need to coordinate teams responsible for:
- Data Engineering — ingestion pipelines, transformations, document processing, metadata and data quality.
- AI/ML Engineering — embeddings, retrieval strategies, model selection, evaluation and optimization.
- Cloud and Infrastructure — compute, networking, storage, scaling, reliability and cost.
- Cybersecurity — identity, authorization, encryption, data protection and monitoring.
- Application Development — user interfaces, APIs, workflow integrations and enterprise applications.
- Compliance and Legal — regulatory requirements, privacy, retention and acceptable-use policies.
- Business Teams — authoritative content, use cases, acceptance criteria and business outcomes.
- Operations — monitoring, incidents, support, change management and continuous improvement.
That is why RAG should be treated as a program, not simply a technical feature.
The Program Manager's Role in Enterprise RAG
For program managers, RAG initiatives create a new set of cross-functional dependencies. Someone needs to coordinate questions such as:
- Who owns the source data?
- Who determines which documents are authoritative?
- How frequently are embeddings refreshed?
- How are user permissions enforced?
- Who defines acceptable retrieval quality?
- How are models and prompts versioned?
- What happens when a source document changes?
- How are incorrect answers reported and investigated?
- Who approves new knowledge sources?
- What metrics determine whether the solution is ready for production?
- Who owns the solution after implementation?
These are not exclusively AI engineering questions. They are program governance questions. And this is one reason AI transformation creates such an important opportunity for experienced program leaders. Many of the hardest enterprise AI problems are not purely technical. They involve ownership, governance, dependencies, risk, operating models, adoption, measurement, and organizational change. Those are familiar program-management challenges appearing in a new technological environment.
What Program Managers Should Measure
Once a RAG application enters production, measuring whether employees "like the chatbot" is not enough. Organizations need metrics across the technical, operational, user, and business layers. Important measures include:
- Retrieval Accuracy - Did the system retrieve the information required to answer the question?
- Answer Relevance - Did the response actually address what the user asked?
- Groundedness - Are the claims in the answer supported by the retrieved enterprise information?
- Citation Accuracy - Do the citations actually support the statements associated with them?
- Latency - How quickly does the system retrieve information and generate an answer?
- Cost per Interaction - What does each query cost once the system operates at enterprise scale?
- User Adoption - Are employees actually using the solution—and returning to it?
- Resolution or Completion Rate - Are users successfully completing the task they came to the AI system to accomplish?
- Business Impact - Is the application improving a measurable organizational outcome?
That could mean:
- Reducing support resolution time.
- Accelerating employee onboarding.
- Reducing time spent searching for information.
- Improving policy compliance.
- Reducing operational errors.
- Increasing analyst productivity.
- Shortening project decision cycles.
The last category is especially important. The original RAG framework emphasizes that a technically impressive system that does not improve a business outcome is still not a successful transformation.
Moving From RAG Pilot to Enterprise Platform
A proof of concept might begin with a relatively simple use case.
- Upload several hundred documents.
- Generate embeddings.
- Store them in a vector database.
- Connect an LLM.
- Build a chat interface.
- Demonstrate that employees can ask questions.
That may prove technical feasibility. But production introduces an entirely different set of questions.
- Can the architecture scale to thousands of users?
- Can it handle millions of documents?
- How will new information be ingested?
- How will permissions be synchronized?
- How will retrieval quality be monitored?
- How will costs be controlled?
- What happens if the LLM provider is unavailable?
- How will incidents be handled?
- How will models, prompts, embeddings, and retrieval configurations be versioned?
- How will employees report incorrect answers?
- How will the organization determine whether the system is producing measurable value?
This is where many AI initiatives encounter the gap between successful demonstration and sustainable enterprise capability. The technology is only one part of crossing that gap. The rest is program execution.
The Bigger Picture: RAG Is Really Knowledge Architecture
RAG represents an important shift in how organizations use generative AI. Instead of relying entirely on what an LLM learned during training, enterprises can provide controlled access to relevant organizational knowledge at the moment a user asks a question. But the model is only one component. The complete architecture is closer to:
Enterprise Knowledge → Governance → Ingestion → Chunking → Embeddings → Vector Database → Retrieval → Context → LLM → Grounded Response → Monitoring → Feedback
Every component affects the quality of the final answer. And every component introduces decisions involving architecture, security, governance, ownership, performance, cost, compliance, and operational support. That is why enterprise RAG cannot be treated solely as a technology implementation. It is a program transformation.
The organizations that succeed with RAG will not necessarily be those using the largest LLM, the newest vector database, or the most sophisticated AI platform. They will be the organizations that build a trusted knowledge architecture around the model. They will know which information is authoritative. They will control who can access it. They will continuously evaluate retrieval and response quality. They will monitor cost and performance. They will establish ownership and governance. And, most importantly, they will connect the technology to measurable business outcomes.
The competitive advantage will not simply come from having an LLM. It will come from giving that LLM access to the right knowledge, at the right time, for the right user, under the right controls.
For program managers leading enterprise AI transformation, understanding that architecture may soon be just as important as understanding cloud migrations, Agile delivery, cybersecurity, or DevOps. Because as generative AI moves deeper into enterprise operations, the question will increasingly shift from: "Which AI model should we use?" to: "How do we build the trusted enterprise architecture around it?"
And that is where program leadership becomes critical.
About the Author
Kimberly Wiethoff, MBA, PMP, PMI-ACP is an AI Transformation Program Leader specializing in enterprise digital transformation, AI-enabled delivery, PMO leadership, AI governance, Agile program execution, cloud transformation, and complex portfolio delivery.
Through Managing Projects the Agile Way, she shares practical strategies for modernizing project and program delivery, strengthening PMO leadership, and preparing organizations and leaders for the future of AI-enabled program management.
#ArtificialIntelligence #AILeadership #ProgramManagement #ProjectManagement #AITransformation #DigitalTransformation #PMO #AIGovernance #AgenticAI #PredictiveAnalytics #EnterpriseTransformation #TechnologyLeadership #ExecutiveLeadership #FutureOfWork #Leadership
Download Document, PDF, or Presentation
Author: Kimberly Wiethoff, MBA, PMP, PMI-ACP