When I wrote this in December 2023, the free ChatGPT's knowledge cutoff date was years before in September 2021. It doesn't know about any news that happened after that, which can cause big problems if you ask it factual questions without understanding how the knowledge cutoff works. If you're like me, then you probably have a few friends and relatives who have mentioned using the free ChatGPT for things like medical advice, which is a pretty bad idea.

But some AI agents can tell you about things that happened after the knowledge cutoff date. They can do web searches and look up stuff in databases. So, how does that work? How can an AI agent look up the status of an order if the AI model's pre-training didn't include that?

Some AI agents can do more than look up stuff, they can do things for you. When you ask ChatGPT to make an image with Dall-E, how do they do it? Is the AI model deeply integrated with the image-generation model so that it returns images along with its text completions?

Not really. The answer is simpler, and it's a preview of a new type of software solution.

Give the AI model reference material

Imagine you’ve got a research assistant. Any time you ask them a question, they can find you the right answer. They don't just happen to know the answer to every question. They just know how to do research and report back to you. They're super-efficient, so if you ask a question then they can research and respond based on what they found in the blink of an eye.

That's how AI agents like Bing, Perplexity, Google's Bard, and ChatGPT with web browsing all look up stuff for you: You ask a question, they go out and do research, and then they summarize what they found for you.

How does the AI model look stuff up?

But how does that work? How does the AI model look up stuff for you? Is there some information-retrieval system deeply wired into the AI model that it uses to look up current information as it's thinking?

No. It doesn't happen deeply within the AI. The answer to all these questions is simple and elegant: The application that uses the AI looks up the stuff and sends the answer to the AI model along with the question. The AI model gets the question and the answer at the same time, and all it does it summarize. The application that uses the AI model does all the lookup work, and the AI model only does what it's good at: writing sentences.

The sequence

Here's what the process looks like. Normally, a question and answer would look like this:

The agent relays the user's "Hello" message to the model, the model responds, and the agent displays the message for the user in whatever chat UI they're using. The business system is not involved.

When we add the ability to look up stuff, the sequence looks a little different. The application first looks up the stuff and then sends it to the model with the question. Here's an example of the user asking for the status of an order, which will require a lookup:

The AI model tells you the name of the function to run and the parameters to use. The agent must execute that function, then report back to the agent API with results.

You end up calling the API two times instead of once. You “augment” the second API request with the results of the function call that you perform yourself.

A basic request and response

Let's see what the API requests and responses look like, without any code. Here's what the simple request from the agent application to the OpenAI API looks like in the first sequence above:

The OpenAI API sends a response that includes the next message in the conversation history, from the AI agent. Those are labeled with the role "agent".

Now let's give it tools

As the application, we're responsible for managing the conversation history. OpenAI doesn't maintain any sort of session state on their side. Every time we send the next request in a conversation to the OpenAI API, we must provide the conversation history. We do that by taking the previous conversation history and adding the latest response from the AI, and the latest message from the user.

This time, we'll also give it a couple of functions that it could call if it decides that it should. We add a list of those functions to the request along with the conversation history. Each function is a description to the AI model that tells it the name of the function, what it's for, and how to call it. Different functions require different parameters, and you use this description to tell it that when it wants to look up an order ID, it needs to provide an order ID. Or that when it wants to create a new order, it needs to provide the customer information and the order items.

There's a structured format (JSON) for describing the functions, but let's skip the code in this simplified diagram for the second sequence above:

When the OpenAI API gets that request, what's in the request is the only thing it knows about the conversation history.

The agent asks you to call a function

So, it looks at the situation and concludes that the user is asking for the status of order ID 123456789. It looks at its list of available functions and sees a function called "check order status" that requires an order ID. Hey, look! We have an order ID! The planets are aligning!

The AI model will check on that order ID by responding like this:

As the agent application, performing the task is now your job. You must execute that function. Your code then queries your business system to get raw order details:

orderId: 123456789
status: 046-IN-TRANSIT
lastUpdate: 1648762800000
location: OAKLAND, CA

Great, you have some raw data. But you can't show that ugly stuff to the user.

Use the AI to describe the results

To generate an AI response to the user, you send a second API request to the OpenAI API. Like before, you're sending a conversation history with the request, and that's the only way the AI model knows anything about what's happening. In that history, there's a message from when the user asked a question, a note that the model decided to call a function, followed by the function call results.

The AI model will take the value that you gave it and use that to generate the next message in the conversation:

Division of responsibilities

Managing the conversation history is our prerogative as the application, which gives us Orwellian control over the past and the present. We can use that to insert things like the status information for an order, proprietary facts, or anything potentially useful for the model as it generates responses.

We're also executing the functions, eliminating concerns about the complexities of deploying code into the OpenAI API to run it.

We're doing all the real work ourselves when we use function calling, but we're using the reasoning ability of the AI model to decide what tools to use, when to use them, and how to use them.

The full application stack for an AI agent looks like this:

In this diagram, an existing business system (bottom left) now supports a new AI agent user interface, managed by the AI agent application. The AI agent code runs separately from the business system and integrates with it through API calls, database queries, or whatever makes sense. The agent uses the large language model (LLM) to generate text for the conversation, using function calling to access the existing software system.

See it for yourself

Go and create an OpenAI secret key and open this Colab notebook, where you can see it in action.

Cost

The only costs will be for the OpenAI API, and we're going to use GPT 3.5 Turbo, so it's cheap. You would have to work hard at it to spend more than a few pennies on this demo.

Requirements

• OpenAI API key: Visit your API keys page to configure an API key that you'll use for the requests.
• Google Colab account [FREE]: Create a free Colab account if you don't already have one. This notebook is so simple that you can run it for free.

Setup

First, open the (new) "secrets" panel in Colab and create a new secret called openai, with your secret OpenAI API access token as the value.

Then, run the first cell of the notebook to use pip to install the OpenAI SDK for Python.

Now you're ready to run the code.

Run it

Run the second cell in the notebook and it will use the secret that you set up to send a request to the OpenAI API for GPT 3.5 Turbo. Note that the code provides a function list with a check_order_status function.

As we discussed above, you don't send an actual function to OpenAI. You don't send them any code. You describe the functions that exist:

    function_list = [
      {
        "name": "check_order_status",
        "description": "Check the status of an order using its ID",
        "parameters": {
          "type": "object",
          "properties": {
            "order_id": {
              "type": "string",
              "description": "The order ID."
            }
          }
        }
      }
    ]

The AI model will see that, and it will keep in mind that it has a function that takes an order ID for checking the order status. If anyone asks for something like that, then it knows how. It's like a contract you've made with the AI about how to handle user requests: a standing promise that the tool will be there when it's needed.

You'll see from the output in the notebook that the first response from the AI will tell your code to run a function. Your code runs a (dummy) function and then sends the second request to the AI model. And then, the model responds by generating text you can relay to the user.

More information

You can find more examples of function-calling with the OpenAI API in the OpenAI Cookbook.

Not just a friendly voice

To wrap it up, think of the AI agent as a super-helpful airline desk assistant, but with some extra tricks up its sleeve. It's not just there for a chat – it can quickly handle tasks like booking travel or finding flight details, faster than even the fastest-typing airline staff. This method requires an additional step, but the payoff is substantial: a live connection to the real world. That's the definition of an AI agent: an AI model that can do things for you, not just talk about them.

The artificial intelligence industry is increasingly focused on integration since an AI agent is much more valuable if it can do things you care about in the systems you already use. One of the key ways to integrate your existing systems with the latest and greatest AI models is to give them a toolbox for accessing your system.

We can help

We have experience integrating enterprise business systems with AI models to create AI agents that can report to you about metrics, notify you about alarms, or trigger actions in your system like creating orders or other business use cases. We also have experience reliably operating mission-critical software features over the long term. If you're looking for help with something like that, please get in touch.

References