Seite wählen

Microsoft Copilot: Testing the capabilities of Generative AI

von Thomas Walder, Thomas Neher und Richard Doblinger | 21. März 2024 | Allgemein, English, Künstliche Intelligenz

Thomas Walder

Senior Consultant

Thomas Neher

Senior Consultant

Richard Doblinger

Senior Consultant

Generative AI (as defined by ChatGPT):

Generative AI refers to algorithms capable of creating new content, such as images, text, or audio, based on patterns learned from existing data. It enables machines to generate novel outputs that resemble human-created content, often used in various creative and practical applications.

Our Mission

Since the first public availability of ChatGPT using AI assistants in our day-to-day work has been a hot topic. In October 2023 Microsoft announced the introduction of generative AI capabilities to Microsoft Copilot Studio. The promise is to provide an easy End-2-End solution for implementing, customizing, deploying, and running chatbots with generative AI capabilities for specific corporate contexts. With that context we wanted to validate this promise by applying Copilot to a real corporate world use case: Find and interpret documents dozing in the deep world of internal file shares by using natural language.The idea was to create an additional way of searching on the internal file share (focusing on documents and slide decks) by talking to a chatbot using natural language. Going into the project we had two main goals:

  • Providing a working chatbot, which can be used to search for documents using natural language (ideally in a production ready setup).
  • Clarify possible architecture, governance, permission management, cost and licensing using Microsoft Copilot Studio.

What is Microsoft Copilot

Currently there is a lot of movement around Microsoft Copilot and Power Virtual Agents (established chatbot solution from Microsoft) with the last major update just being released [1]. Therefore, it is even more important to get an overview about what “Copilot” actually is. You should be aware of following major distinctions:

  • In general Copilots are Microsoft’s artificial intelligence companions. Now there are different Copilots competing for completely different tasks.
  • The term Microsoft Copilot itself refers to Power Virtual Agents, just under a new name. What was previously known as Power Virtual Agents got rebranded as Microsoft Copilot with the latest Microsoft Ignite conference in November 2023.
  • Several Copilots are spread across Microsoft’s application universe, such as Microsoft 365 (“Copilot for Microsoft 365”), Copilot for Service, GitHub Copilot to name just a few.
  • Microsoft Copilot Studio refers to the graphical environment enabling users to build standalone chatbots using Generative AI.
  • “Copilot” is also used as a term for chatbots created with Microsoft Copilot Studio.

All this movement around made it quite hard to lift the fog around the different terminology. We also learned early that the provided documentation currently does not always keep track of the changes and must be used carefully. In the following we use the term “Copilot” to describe chatbots created with Microsoft Copilot Studio.

Architecture

To get an overview describing the architecture, we focus on the functional components of Microsoft Copilot Studio and the new functionalities compared to Power Virtual Agents. For our purposes it is sufficient to use a simplified architecture of Microsoft Copilot Studio as shown in Figure 1 (a more detailed view can be found in the implementation guide from Microsoft[2]):

Figure 1: Architecture of Microsoft Copilot Studio

Microsoft Copilot Studio (1) is a graphical development environment to build, deploy and manage Copilots. A Copilot consists of multiple topics (2), which are conversation paths for the chatbot. For example, the topic “Greet User” gives the bot a script for how to react if a user writes “hello”. Topics are implemented via a graphical workflow (see below for an example).

These rule-based topics (3) are known from Power Virtual Agents, which is Microsoft’s established product for building Copilots. Power Virtual Agents is currently being merged into and rebranded as Microsoft Copilot.

With the new type of topic “Generative Answers” (4) conversational AI capabilities are introduced. This type of topic enables users to interact with the Copilot using natural language and gives the bot the possibility to generate answers even if no conversational path is implemented. Results are provided by using Azure AI Search and Azure OpenAI Service.

These answers are based on the provided data sources (5), which are configured during creation of the Copilot. Data and files from these sources are used as input for generating the answer to a user request.

Currently there are four possible data sources [3]:

  1. Public websites
  2. Internal SharePoint and OneDrive (up to different 4 sources)
  3. Manually uploaded files (up to 3 MB)
  4. Custom data in JSON format (up to 3 MB)

After configuring the topics and data sources, the Copilot can be published to one or more channels (6). The easiest way to a working Copilot is via using the custom site option, which provides an embeddable link to a working chatbot. Other than that there is a wide array of possible integrations into other tools available e.g. Teams. Other functionality (7) includes built in monitoring of Copilot & operational support (for example one-click deployments and authentication services).

Implementation, Demonstration and Evaluation of our Copilot

Our use case was to find and interpret documents dozing in the deep world of internal file shares by using natural language. So as a user I want to be able to ask questions about documents (e.g. “for what customers did we implement a data warehouse”) and the answer consists of the relevant documents with a short summary. This can then be the starting point for creating new pitch documents or to identify knowledge carriers.

The setup process for the chatbot is straightforward:

Figure 2: Creation of a new copilot in Microsoft Copilot Studio

All that is needed for implementation is a name, a language for the chatbot and a SharePoint site. For testing purposes, we collected several documents (Word, PowerPoint, Excel …) in a SharePoint site. The specific number of uploaded presentations is not crucial as the bot will return the corresponding documents, when the right questions are asked and if they are present. We connected our bot to that SharePoint site by specifying its URL. As a best practice, Microsoft recommends omitting the leading ‘https://’. Further sites or public websites can be specified later in the process, if needed. Upon clicking “Create”, the initial demo bot is ready to be used. When creating bots one can now input questions and receive responses from the bot. These responses contain summaries of relevant documents and are returned along with corresponding citations. We found that questions usually work best when they are concise and ask for a summary of the documents’ content. For example, when asked “What is low code”, the bot retrieves relevant documents and summarizes their content as follows:

Figure 3: Exemplary Pitch Copilot interaction

Sometimes, the bot is sensitive to the precise way the questions are asked. For example, the bot couldn’t process the question “For which customers have we already developed a front end?”. However, the formulation “can you find something about front end development” yields an informative answer. This is a result of missing meta information, such as to whom the word “we” in the question is referring to.

Figure 4: Second example of a Pitch Copilot interaction

Consistency is another limitation of the bot. When asked the same question several times, it can return varying results due to its wide array of possible unique outputs. It also means that the bot doesn’t necessarily yield all relevant documents in one answer. For a more exhaustive result, the bot should be asked to provide documents several times.

Figure 5: Sensitivity of Pitch Copilot to language

We tested the bot with both English as well as German prompts. Overall, we noticed that the bot performed better when asked in German and sometimes didn’t understand the English prompts. We attribute this behaviour to the fact that the entirety of documents uploaded on SharePoint are in German. We also set up a Copilot which uses a whole SharePoint instead of a sub site. In that case only the documents are considered, for which the user is authorized as per AD-permissions. The results were consistent with our expectations. In a test scenario, where the bot was asked to find documents related to data warehouses, we identified three relevant documents, all of which were successfully retrieved by the bot. To get the bot into production for other users, it must be published in a designated step. For this, an authentication method must be implemented. This ensures that the search only contains the resources that a given user has access rights to. There are three authentication methods for the bot:

Figure 6: Copilot authentication methods

As of now, backing the bot with SharePoint data is only possible with manual authentication with Microsoft Entra ID. For this, an app registration must be manually set up in the Azure Portal. After the registered bot has been published, a link to the bot can be distributed to other users in the same tenant, who then have to login in via SSO. In summary, setting up the bot is a simple, one-step process. Publishing it requires more effort but is also straightforward and robust. The resulting bot could retrieve relevant documents which works best when questions are short and not too specific. If the bot answers the documents returned match the question accurately, but the answers are not always exhaustive or consistent.

Licensing & Billed Sessions

To get started with authoring a Copilot and to get access to Microsoft Copilot Studio on organizational level you need to purchase a tenant license for 200$ per tenant/month.

This enables administrators in the admin portal to assign Microsoft Copilot Studio user licenses to individual users within your tenant at no additional costs or limits. A granted user license allows you to manage, author and publish bots within Microsoft Copilot Studio for your organization. After publishing a Copilot the users do not need to purchase a specific license but can access and interact with the bot directly within the boundaries of your configured user authentication scheme[4].

The tenant license includes 25,000 messages/tenant/month, where a message represents the common currency across the platform.

In simple terms a message is described as a request sent to the Copilot which triggers an action. Depending on the request different consumption rates apply. Meaning “simple” , rule based, requests get billed as one message, whereas requests answered by using Generative AI are billed as two consumed messages. Let’s sketch an example:

  1. “What are opening hours of store XY?” – 1 message
  2. “What are store’s XY opening hours the upcoming holidays?” – 2 messages

Another way to get started is to make use of rights with select Microsoft/Office 365 Licenses. This enables you to publish Copilots within Microsoft Teams only and since February 2024 within a Microsoft 365 licence. For Details see the current licencing guide[4].

Data Governance & Security

Microsoft announced availability of Generative AI features in Europe beginning of December 2023. This enables Copilots using Generative AI within the boundaries of the European Union (which is a must to comply with most European organizational data governance guidelines). It is ensured by Microsoft that data storage under the hood of Azure OpenAI stays within European data centers [5]. However, the Bing Search must be disabled within the environment configuration to ensure no data is transferred to the US, which is a crucial drawback for your Copilot’s functional scope.

Summary and key Takeaways

In general, reaching our main goal of implementing a working chatbot capable of delivering useful GenAI based answers was quite easy and fast. We therefore were able to invest more time into the second goal of providing an overview of architecture, governance, permission management, cost and licensing of Microsoft Copilot.

Our key takeaways based on the current maturity level of Microsoft Copilot are:

  1. Microsoft Copilot Studio offers an easy-to-use way of creating AI based chatbots with little to no technical knowledge.
  2. The use case “search for relevant documents with natural language” is especially easy and only limited by how many documents are available to the specific user.
  3. The rebranding of Power Virtual Agents into Microsoft Copilot Studio and the high pace in change make it difficult to find relevant information in the available documentation.

References

[1] Medhi, Y. (2023). Celebrating the first year of Copilot with significant new innovations. Microsoft Blogs. https://blogs.microsoft.com/blog/2023/12/05/celebrating-the-first-year-of-copilot-with-significant-new-innovations/. [18 January 2024]

[2] Jammes, H. (2023). New Microsoft Copilot Studio implementation Guide. https://microsoftcopilotstudio.microsoft.com/en-us/blog/new-microsoft-copilot-studio-implementation-guide/. [19 January 2024]

[3] Microsoft. Generative answers with Search and summarize content. https://learn.microsoft.com/en-us/microsoft-copilot-studio/nlu-boost-node. [18 January 2024]

[4] Microsoft. Power Platform Licensing Guide February 2024. p. 18f. https://go.microsoft.com/fwlink/?linkid=2085130. [19 Februrary 2024]

[5] Microsoft. Geografische Regionen zur Datenspeicherung und -verarbeitung. https://learn.microsoft.com/de-de/microsoft-copilot-studio/manage-data-movement-outside-us.[19 January 2024]