Users often struggled to find relevant documentation and had difficulty comparing services effectively. To address this, we built a search bot designed to help users locate information quickly and easily. This bot streamlines the search process and resolves the confusion, providing a more efficient way for users to access the documentation they need.

For instance, here is the flow to get information on all MCS services:

We got a lot of negative feedback for the last arrow…

This situation has motivated the team to innovate and suggest a modern solution. And we are now very happy to present this new feature: the MCS SearchBot!

For the same question, the flow is now:

How to Get Started?

There is only one chatbot for all documentations, and it is accessible from all of our websites.

First, simply go to any online documentation listed here:

• apim: https://doc-apim.mcs.thalesdigital.io/docs
• dbaas: https://doc.dbaas.thalesdigital.io/docs/
• iaas: https://doc-iaaslz.mcs.thalesdigital.io/docs
• iam: https://doc-iam.mcs.thalesdigital.io/docs/
• k8saas: https://doc.kaas.thalesdigital.io/docs/
• network: https://doc-network.mcs.thalesdigital.io/docs/
• vdi: https://doc-vdi.mcs.thalesdigital.io/docs

Second, click on this logo:

Then, you should see:

How it works?

It works with Generative Artificial Intelligence (AI)!

Under the hood we used a RAG: Retrieval-Augmented Generation.
RAG is a complex term for a simple idea: include relevant information (documentation) with the user query into a big prompt and let the AI figure out the answer!

To achieve this result there are two different stages:

• indexation: convert the documentation into semantic vectors to grasp the meaning of the document pages
• retrieval: use the user query to get the relevant documentation vectors and send everything to the AI model.

The idea is that new general AI models (like ChatGPT) are getting better so fast that it is pointless to train our own model. Instead, let’s build a modular series of tasks to leverage its value and be able to replace the model at any point with the next version.

The next two stages will be more technical! Get ready to dive in! 😉

Indexation

The first step is to split our documentation into chunks, transform and store them into vectors.

💡 info: The vectors we are using are embeddings. An embedding is a multi-dimensional vector that represents the meaning of the word. For example, a “dog” embedding will be close to “cat” and far away from “nuclear.” It will allow us to easily find the most meaningful document to the user query by searching the closest embedding.

The indexation process is in a GitLab CI/CD pipeline:

• (1) Only the documentation raw text pages are selected (markdown in .md and .mdx).

• (2) Every documentation page is split into chunks according to the sections and titles.

  Technical details: Leveraging Markdown semi-structured data, we split based on headers using the Python library LangChain. The chunks that are too short are merged together to only keep chunks with at least 216 tokens (1 token ~= 4 characters). The size matters according to the embedding model used at the next step.

• (3) Each chunk is transformed into a vector using an AI model accessed through the Trustnest GenAI APIM (the secure apim service we provide).

  Technical details: We are using the Azure OpenAI embedding model “text-embedding-ada-002”. This model is known to efficiently capture more context but needs larger text chunks to be efficient.

• (4) The final data is sent to our vector database (Azure Search) using the fields:

  • mcs_service: the documentation service name (ex: k8saas)
  • file_path: the markdown file path in the documentation
  • header: the markdown header for the chunk
  • content: the embedding vector
  • raw_content: the raw text content of the chunk
  • id: unique UUID made from the header and file_path

💡 info: The advantage of using CI/CD pipelines, is that the indexation process is mutualized for all the documentations, and that any update made to a documentation page triggers an automatic update in the vector database.

Retrieval

Now that all of our documentation is indexed, let’s use it to answer the users’ queries!

The goal is to get relevant documents according to the user query to make a prompt with all the information in it. Then this prompt is sent to an AI model (the LLM GPT-4o):

There are two parts: the Chatbot User Interface (UI) sends the user request to our MCS Chatbot API.

• The Chatbot UI: an interface in all the documentation websites that calls the API with:

  • A session ID stored in the browser cookies
  • The user question
  • The user feedback

• Then in the Chatbot API (developed in Python):

  • (1) Request the vector database with the user query to get the relevant documentation chunk

    Technical details: We are using Azure Search “hybrid search”, which searches using both the embeddings and the raw text field. Then only the most relevant documents (with an Azure Search ‘search score’ >= 0.03).

  • (2) Create and send the prompt with:

    • A generic context: “You are an assistant for the Trustnest MCS public documentation, your role is …” (complete prompt below)
    • The user query
    • The conversation history for the current session (gathered from the history database Redis)
    • The relevant documentation extracts

    Then the prompt is sent to the chat model (GPT-4o) via the Trustnest GenAI APIM. And the answer is sent back to the user.

    Technical details: We had trouble with recurrent broken documentation links from the AI. So we decided to include the relevant documentation manually as a footer, reconstructing them using the data from Azure Search: mcs_service + file_path + header. This also has the benefit to simplify the prompt.

  • (3) Updates conversation history in the history database (Redis).

  • (4) Saves query/answer and feedback in the monitoring tool (Langfuse: https://langfuse.com/docs).

In the end, the prompt we are sending the chat model is:

You are an expert assistant helping users with documentation related to Trustnest Managed Cloud services (MCS) services. Your task is to provide clear and concise answers to user queries based on the relevant documentation pages provided.

Service Descriptions:

• K8SaaS: Trustnest Managed Kubernetes (k8saas) provides Azure Kubernetes Services Clusters as a service. It aims to deliver production-ready clusters with transversal services such as Observability, Security, Access Management, Performance, Storage, and Cost Optimization.
• IaaS: Trustnest Infrastructure Landing Zones (iaas) provides landing zones based on an Azure subscription with the “production” option enabled by default. These are preconfigured cloud environments considering scale, security, governance, networking, and identity.
• IAM: Trustnest Identity & Access Management (IAM) provides managed services concerning authentication and RBAC issues on Azure for iaas landing zones and k8saas clusters.
• DBaaS: Trustnest Managed Databases (DBaaS) provide PostgreSQL and CosmosDB databases on Azure to simplify system development while optimizing costs.
• Network Backbone: Trustnest Network Backbone provides managed services concerning network issues on Azure for iaas landing zones and k8saas clusters.
• VDI: Trustnest Virtual Desktop Infrastructures based on VMware Horizon Cloud on Microsoft Azure. Extends the existing corporate laptop access point to access Trustnest MCS services.
• APIM: TrustNest has launched an APIM to provide a single place to list and access Trustnest internal APIs (limited only to REST - for now). It uses Azure APIM and enforces MFA.

Instructions:

1. Use the user’s query, the provided relevant documentation sections, and the conversation history to generate a clear and concise answer that directly addresses the query.
2. If the query is unrelated to Trustnest services or the answer is not explicitly in the documents, say that you don’t have the answer in the documentation, and give a hint based on your knowledge.
3. If you found any relevant post-it links (starting with ‘https://postit .service-now.com/esc’) please mention it at the end of your answer in a section “Ticketing links”.
4. Do not, Never ever ever, provide any links to markdown files. They will be incorrect as you don’t have the base URLs. I’m providing them directly to the user.

User Current Question: {query_text}

Chat History: {history}

Relevant Documents: {relevant_documents}

Your Answer:

The tricky part is to be specific enough to have relevant answers but not too long or complicated to generate less precise answers. Using this prompt instead of a generic one improved the chatbot answers drastically !

User feedback

For every answer in the chatbot, the user can like, dislike, and give feedback on the prompt:

Then in our monitoring tool Langfuse, we can list those feedback, see the original query and response, and even list all of the conversations related to a session.

Every user feedback is carefully reviewed for the project to be a virtuous cycle where feedback on chatbot answers improves the documentation, and better documentation improves the chatbot answers!

Next Steps

First, we want to closely track the usage of this new service to ensure it effectively meets your needs. Please evaluate the searchbot answer, and if possible give us a comment to help improving this tool.

The next major technical steps will be:

• Including links to relevant site pages like Microsoft Learn or Kubernetes documentation.
• Including calls or references to our services’ APIs directly from the chatbot.
• Add the chatbot to other teams within Thales by plugging it to the documentation repo and updating the prompt context.

Thanks for reading!

Take care !