This tutorial will walk you through using the Azure OpenAI embeddings API to perform document search where you'll query a knowledge base to find the most relevant document.

In this tutorial, you learn how to:

Important

We strongly recommend using text-embedding-ada-002 (Version 2). This model/version provides parity with OpenAI's text-embedding-ada-002. To learn more about the improvements offered by this model, please refer to OpenAI's blog post. Even if you are currently using Version 1 you should migrate to Version 2 to take advantage of the latest weights/updated token limit. Version 1 and Version 2 are not interchangeable, so document embedding and document search must be done using the same version of the model.

If you haven't already, you need to install the following libraries:

BillSum is a dataset of United States Congressional and California state bills. For illustration purposes, we'll look only at the US bills. The corpus consists of bills from the 103rd-115th (1993-2018) sessions of Congress. The data was split into 18,949 train bills and 3,269 test bills. The BillSum corpus focuses on mid-length legislation from 5,000 to 20,000 characters in length. More information on the project and the original academic paper where this dataset is derived from can be found on the BillSum project's GitHub repository

This tutorial uses the bill_sum_data.csv file that can be downloaded from our GitHub sample data.

You can also download the sample data by running the following command on your local machine:

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

After setting the environment variables, you may need to close and reopen Jupyter notebooks or whatever IDE you're using in order for the environment variables to be accessible. While we strongly recommend using Jupyter Notebooks, if for some reason you cannot you'll need to modify any code that is returning a pandas dataframe by using print(dataframe_name) rather than just calling the dataframe_name directly as is often done at the end of a code block.

Run the following code in your preferred Python IDE:

The output of this command will vary based on the number and type of models you've deployed. In this case, we need to confirm that we have an entry for text-embedding-ada-002. If you find that you're missing this model, you'll need to deploy the model to your resource before proceeding.

Now we need to read our csv file and create a pandas DataFrame. After the initial DataFrame is created, we can view the contents of the table by running df.

Output:

The initial table has more columns than we need we'll create a new smaller DataFrame called df_bills which will contain only the columns for text, summary, and title.

Output:

Next we'll perform some light data cleaning by removing redundant whitespace and cleaning up the punctuation to prepare the data for tokenization.

Now we need to remove any bills that are too long for the token limit (8192 tokens).

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Create and assign persistent environment variables for your key and endpoint.

For this tutorial, we use the PowerShell 7.4 reference documentation as a well-known and safe sample dataset. As an alternative, you might choose to explore the Microsoft Research tools sample datasets.

Create a folder where you would like to store your project. Set your location to the project folder. Download the dataset to your local machine using the Invoke-WebRequest command and then expand the archive. Last, set your location to the subfolder containing reference information for PowerShell version 7.4.

We're working with a large amount of data in this tutorial, so we use a .NET data table object for efficient performance. The datatable has columns title, content, prep, uri, file, and vectors. The title column is the primary key.

In the next step, we load the content of each markdown file into the data table. We also use PowerShell -match operator to capture known lines of text title: and online version:, and store them in distinct columns. Some of the files don't contain the metadata lines of text, but since they're overview pages and not detailed reference docs, we exclude them from the datatable.

View the data using the out-gridview command (not available in Cloud Shell).

Output:

Next perform some light data cleaning by removing extra characters, empty space, and other document notations, to prepare the data for tokenization. The sample function Invoke-DocPrep demonstrates how to use the PowerShell -replace operator to iterate through a list of characters you would like to remove from the content.

','

After you create the Invoke-DocPrep function, use the ForEach-Object command to store prepared content in the prep column, for all rows in the datatable. We're using a new column so the original formatting is available if we would like to retrieve it later.

View the datatable again to see the change.

When we pass the documents to the embeddings model, it encodes the documents into tokens and then returns a series of floating point numbers to use in a cosine similarity search. These embeddings can be stored locally or in a service such as Vector Search in Azure AI Search. Each document has its own corresponding embedding vector in the new vectors column.

The next example loops through each row in the datatable, retrieves the vectors for the preprocessed content, and stores them to the vectors column. The OpenAI service throttles frequent requests, so the example includes an exponential back-off as suggested by the documentation.

After the script completes, each row should have a comma-delimited list of 1536 vectors for each document. If an error occurs and the status code is 400, the file path, title, and error code are added to a variable named $errorDocs for troubleshooting. The most common error occurs when the token count is more than the prompt limit for the model.

You now have a local in-memory database table of PowerShell 7.4 reference docs.

Based on a search string, we need to calculate another set of vectors so PowerShell can rank each document by similarity.

In the next example, vectors are retrieved for the search string get a list of running processes.

Finally, the next sample function, which borrows an example from the example script Measure-VectorSimilarity written by Lee Holmes, performs a cosine similarity calculation and then ranks each row in the datatable.

The commands in the next example loop through all rows in $Datatable and calculate the cosine similarity to the search string. The results are sorted and the top three results are stored in a variable named $topThree. The example does not return output.

Review the output of the $topThree variable, with only title and url properties, in gridview.

Output:

The $topThree variable contains all the information from the rows in the datatable. For example, the content property contains the original document format. Use [0] to index into the first item in the array.

View the full document (truncated in the output snippet for this page).

Finally, rather than regenerate the embeddings every time you need to query the dataset, you can store the data to disk and recall it in the future. The WriteXML() and ReadXML() methods of DataTable object types in the next example simplify the process. The schema of the XML file requires the datatable to have a TableName.

Replace with the full path where you would like to write and read the XML file. The path should end with .xml.

As you reuse the data, you need to get the vectors of each new search string (but not the entire datatable). As a learning exercise, try creating a PowerShell script to automate the Invoke-RestMethod command with the search string as a parameter.

Using this approach, you can use embeddings as a search mechanism across documents in a knowledge base. The user can then take the top search result and use it for their downstream task, which prompted their initial query.

If you created an OpenAI resource solely for completing this tutorial and want to clean up and remove an OpenAI resource, you'll need to delete your deployed models, and then delete the resource or associated resource group if it's dedicated to your test resource. Deleting the resource group also deletes any other resources associated with it.

Learn more about Azure OpenAI's models:

Azure OpenAI Service models