How to visualize Semantic Kernel & Azure OpenAI plans using Mermaid

Using the function calling ability of Semantic Kernel/OpenAI is very exciting from an application development point of view. Instead of hard-coding paths though the system, Semantic Kernel & OpenAI can decide for themselves what plugins to call, the order of operations, parsing input & output, etc.

However, it can be challenging to understand why the system decided to go down a path & all of the operations, prompts, function calls, etc. that occurred.

Semantic Kernel provides a ChatHistory class that records “most” of the calls made, the associated role & the order of operations.

We can use this to generate a SequenceDiagram in Mermaid to visualize the system.

Here is how I visualized my OpenAI call plans for my Semantic Kernel sleeping bag example app (see the previous blog posts about how this example app works).

Mermaid

I used a JavaScript library called Mermaid to generate diagrams on the fly. Mermaid is a diagraming & charting tool that uses a “Markdown”-like text language for defining diagrams. The library then renders the diagram based upon the text. Mermaid also includes a live editor where you can play with the diagram before trying to embed it in your app.

Specifically for OpenAI, I used the “Sequence diagram” since I want to display a timeline of function calls. I tried to differentiate between the various roles, native plugins, semantic plugins, etc that are involved to give the user a sense of how & why the OpenAI model returned the result it did.

Here is a simplified example.

The Mermaid Markdown to generate this diagram is:

sequenceDiagram
    OpenAI->>WebAppSystemYou are a customer support chatbot. You should answer the question posed<br/> by the user. Ground your answers based upon the user's purchase history. Make s<br/>ure and look up any needed context for the specific user that is making the requ<br/>est. If you don't know the answer, respond saying you don't know. Only use the p<br/>lugins that are registered to help you answer the question. <br/> Usernamedkschrute <br/> Current DateJan-10-2024
    Note over OpenAI,WebAppsystem
    WebApp->>WebAppWill my sleeping bag work for my trip to Patagonia next month?
    Note over WebApp,WebAppuser
    OpenAI->>WebAppOrderHistoryPlugin_OrderHistoryLookup - {  "username""dkschrute"}
    Note over OpenAI,WebAppassistant - P:858/C:22/T:880
    WebApp->>OrderHistoryPlugin_OrderHistoryLookup: {  "username""dkschrute"}
    Note over WebApp,OrderHistoryPlugin_OrderHistoryLookupassistant
    OrderHistoryPlugin_OrderHistoryLookup->>WebApp: {"orderId":"1","customerId":"dkschrute","orderDate":"2021-01-01","orderTotal":"1<br/>0","orderStatus":"Shipped","orderItems":[{"productId":"12345","productName":"Eco<br/> Elite Sleeping Bag","productPrice":"10","productQuantity":"1"}]}
    Note over OrderHistoryPlugin_OrderHistoryLookup,WebApptool
    WebApp->>OpenAIOrderHistoryPlugin_OrderHistoryLookup {"orderId":"1","customerId":"dkschrute","o<br/>rderDate":"2021-01-01","orderTotal":"10","orderStatus":"Shipped","orderItems":[{<br/>"productId":"12345","productName":"Eco Elite Sleeping Bag","productPrice":"10","<br/>productQuantity":"1"}]}

Here is the short explanation of the syntax.

  • sequenceDiagram – Tells Mermaid to generate a sequence diagram
  • OpenAI->>WebApp syntax tells the diagram to draw an arrow from one box to another. 
  • Note over syntax helps document each step.

There are a huge number of rendering options, styling, etc, but this is the basic idea.

How to embed Mermaid in your React app

The Mermaid library is a JavaScript library, so import it into your app, then include it in a div. You will write a JavaScript function similar to below to convert the raw chat history data from the API to Mermaid.

The ConvertToMermaidDiagram function loops over the ChatHistory and builds the Markdown string to return.

Here is a link my complete React component that includes the JavaScript that converts to the Mermaid syntax (note that this is still under development and may change over time).

semantic-kernel-sleeping-bag/src/recommendation-web-app/src/components/ResponseGraph/ResponseGraph.tsx at main · jordanbean-msft/semantic-kernel-sleeping-bag (github.com)

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *