Connecting External LLM Endpoint: Azure OpenAI #
Kolena connects with Azure OpenAI to leverage your hosted LLM endpoint to do data enrichment and result evaluation. In this tutorial, we'll learn how to establish an integration between Kolena and your Azure OpenAI deployment.
To get started, ensure you have administrator access within Kolena. Navigate to the "Integrations" tab on the Organization Settings page under the "LLM Integration" section and click "Add Integration", then "Azure OpenAI".
Note: By default, Kolena will make up to 100 concurrent request to your Azure OpenAI deployment to speed things up, please make sure your deployment can handle this load.
Step 1: Create Azure LLM Registration for Kolena#
We will generate an App registration for Kolena in Azure and then assign roles to this registration:
- From the Azure portal, search for "App registrations" and navigate to this page
- Click "New Registration"
- Under "Supported account types", select "Accounts in any organizational directory"
- Click "Register" to save the App registration
- Click on the App registration you have created
- Note the "Tenant ID" and "Application (client) ID"
- Click "Certificates & secrets", then "New client secret"
- Click "Add" to save this secret and note the key value
Step 2: Assign Roles to App Registration#
We will assign a role to the App registration created above:
- Navigate to the openai resource containing your OpenAI llm deployment
- Search for "Access control" on the left panel
- Click "Role assignments"
- Click "Add", then "Add role assignment"
- Search for and select Cognitive Services OpenAI User, this will allow Kolena to make requests to your OpenAI deployment
- Click on the "Members" tab, then click "Select members"
- Search for the App registration created in step 1
- Click "Select"
- Click "Review + assign" to save
Step 3: Save Integration in Kolena#
Return to Kolena and fill in the fields for the Integration and then click "Save".
Field | Description |
---|---|
Tenant ID | The Tenant ID of the App registration created in step 1 |
Client ID | The Application (client) ID of the App registration created in step 1 |
Client Secret | The secret key for the App registration created in step 1 |
Endpoint | The Azure OpenAI endpoint, it can be find under Resource Management/Keys and Endpoint section in Azure OpenAI console |
Model Name | The Name of the model to be displayed on kolena |
Deployment Name | The deployment name for the model to be deployed on Kolena Resource Management/Model Deployment section in Azure OpenAI console |
Support Image | Whether this hosted model has vision capabilities |