Promptmux.ai (prompt multiplexer) is a simple prompt and inference management and experimentation platform. Promptmux.ai serves as a unified middleware between your application and LLM APIs. With Promptmux.ai, you can partition prompts between projects and tasks, serve different infernces and prompts to groups of users and serve multiple prompts for the same task using our simple API. You can also create experiments to A/B test new prompts and models in production.
Promptmux.ai also keeps track of detailed data for you to analyze and make informed decisions and improvements to your LLM configurations. We store generation results side-by-side with served prompts for each of your inference requests. We also store request latency, request statuses, usage statistics, and more. You can access all of this data from your dashboard.
To get started, you'll need to create a free account. Once you have an account, you're ready to create your first project.
Once you have an account, you can login and create a project by clicking on New Project and filling in the resulting modal. Project names are case-insensitive and can only consist of letters, numbers, and underscores. A project represents a group of tasks. You can think of a single project as representing a single LLM powered application.
With a project created, you can start creating tasks for it. A task represents a single use case of a large-language model such as summarization, question-answering, and more. You can create a new task for a project by navigating within the project and clicking on New Task. Tasks are represented with a task key which is used to delineate tasks within a project at request-time. Task keys are case-insensitive and must consist only of letters, digits, and underscores.
Once you've created a task, you can start creating prompts for a task. Promptmux.ai stores prompts as patterns, which are essentially Python style f-strings. You can create as many prompts as you want for a single task, and reference them by ID to receive the prompt you want at request-time.
When you create a prompt pattern, Promptmux.ai will automatically infer the necessary parameters given the pattern specified. The names you used for the variables passed in your prompt pattern should match those provided when making requests to the API.
In addition to creating prompts for a task, you can set a default prompt for a task. A default prompt is the prompt returned by the API when you only specify a task. To set a prompt as a default prompt, simply click on the desired prompt and click the Mark as default button in the resulting modal.
In addition to tasks Promptmux.ai supports inferences. Inferences are just inference configurations from one of our provider APIs. You can create inferences for both generations and embeddings models from OpenAI, co:here, and HuggingFace.
The Promptmux.ai API provides a unified inference experience. For each provider and inference type we return a normalized inference result. You can switch providers at any time without changing your application logic.
When you create an inference, you just need to select a provider, type, and model. For OpenAI and co:here, we support all of their public models. We do not support private models at this time. For HuggingFace, we support any of the public models which are tagged for feature extraction or text generation. We do not support private or public endpoint inference at this time.
Model configurations are specific to the provider and inference type. After you've selected your provider and type, you can customize the configuration options for an inference. These will be the parameters sent at inference time to the provider. You can override any of these options in the API.
You may optionally attach prompts to an inference configuration. This will be the default prompt used for the given inference if no other prompt is specified.
Often times when rolling out a new feature, it's desirable to test the feature on a percentage of users before rolling out to all of them. A common technique for doing this is A/B testing. Promptmux.ai supports A/B testing of prompts and inferences via experiments. You can create an experiment which splits your traffic between 2 prompts or inferences according to specified fractions.
To create an experiment, first create at least 2 prompts or inferences for a task. Then, click on the New Experiment button in the task action menu. Select a prompt or inference depending on the type of experiment you want to create. Finally, configure your experiment in the resulting modal. Once you have your experiment created, you can begin making requests to the experiment API endpoint.
When creating inference experiments, you must ensure you select inference configurations with compatible models.
In some applications, it makes sense to deliver a different set of features to diferent sets of users. With Promptmux.ai, you can create groups for tasks and specify inference and prompt behavior for each group.
Working with groups is simple. Inside a task, click New Group and create a group key. A group key is a unique identifier for a group within a task. Within each group you can specify a prompt and/or an inference. After creating a group, you can request the inference or prompt for a specific group with an additional request parameter.
After some time making requests to the API, you may want to download and analyze the data side-by-side with generated responses to improve your prompts and models. Promptmux.ai supports downloading data for individual tasks as well as for experiments. To download data for a task, simply navigate to the desired task dashboard and click on the download icon on the right side of the task title. To download data for an experiment, click on the desired experiment and click Export data.
Exported data will be in the form of a CSV with the fields access_id, served_prompt_id, prompt, experiment_id. Note that access_id is intended to be a unique identifier for pairing responses with prompts. Access IDs are returned from the API at request time, and should be logged side-by-side with the response for later retrieval. We're working hard on improving our integration with existing LLM APIs to make this process more seamless.
The Promptmux.ai API is a simple REST API. All API endpoints can be accessed at https://promptmux.ai/api/v1/.
The API uses a bearer token authentication to authenticate requests. You must first create an API key by navigating to your settings and clicking Create a new API key. Note that you'll only get to see your API key once, so save it in a safe place.
With an API key, you can authenticate requests by setting the authorization header, like this:
curl -XGET https://promptmux.ai/api/v1/status -H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json"
The Promptmux.ai API is free-to-use. In order to sustain this we support a maximum rate of 60 requests per minute. If you need an increase, please contact support.
If you run into any issues, have feature requests, or just have questions, you can start a discussion here or e-mail support@promptmux.ai.
Return a prompt
If a default prompt is not specified for the given task, you must specify either one of prompt or group. prompt takes precedent over group.
curl -XPOST https://promptmux.ai/api/v1/prompt \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "my_project",
"task": "my_task",
"params": {
"context": "foo bar"
}
}'
{
"data":
{
"access_id": "3e4ac53a-2ad6-4cf7-b690-321bdeb23c2d",
"id":"de53664b-20f8-4954-8159-61575009b063",
"prompt": "The prompt: foo bar"
}
}
Return a list of prompts for the given task
curl -XPOST https://promptmux.ai/api/v1/prompt/list \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "my_project",
"task": "my_task"
}'
{
"data": [
{
"id": "de53664b-20f8-4954-8159-61575009b063",
"prompt": "Use the following pieces of context to answer the question at the end. {context} {question}..."
},
...
]
}
Execute the given inference configuration and return the result
curl -XPOST https://promptmux.ai/api/v1/inference \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "my_project",
"task": "my_task",
"prompt": "hello",
"api_key": "PROVIDER_API_KEY"
}'
{
"data": {
"access_id": "1733e97b-a38a-43b9-8a85-61804f9ad93a",
"id": "949df7a9-8aaf-40d3-9efc-bd73d9dff075",
"model": "medium",
"prompt": "Hello!",
"provider": "cohere",
"raw": {
"generations": [...],
"id": "d8e53599-1d80-4ac1-a5d5-40450eedbd22",
"meta": {...},
"prompt":"Hello!"
},
"results": {
"generations": [" My name is Xiaomi, and I'm the leader of the algae from space, they live"]
},
"type": "generation"
}
}
Return a list of inferences for the given task
curl -XPOST https://promptmux.ai/api/v1/inference/list \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "my_project",
"task": "my_task"
}'
{
"data": {
"inferences": [
{
"config": {
"end_sequences": [],
"frequency_penalty": 0.0,
...
},
"id": "56dc2b02-6ac0-45d7-ba1c-b04edcdb877d",
"model": "medium",
"provider": "cohere",
"type": "generation"
},
...
]
}
}
Return one of the prompts from the given experiment
You may use the user parameter to tie the same prompt to the same user during the course of an experiment.
curl -XPOST https://promptmux.ai/api/v1/experiment/prompt \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "my_project",
"task": "my_task",
"experiment": "experiment_id",
"params": {
"parameter_for_prompt1": "foo",
"parameter_for_prompt2": "bar"
}
}'
{
"data": {
"id": "de53664b-20f8-4954-8159-61575009b063",
"access_id": "ee53664b-20f8-5814-4598-61575009b063",
"prompt": "foo"
},
...
]
}
Execute one of the inferences in the A/B test
curl -XPOST https://promptmux.ai/api/v1/inference \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "my_project",
"task": "my_task",
"experiment": "experiment_id"
"prompt": "hello",
"api_key": {
"a": "PROVIDER_A_API_KEY",
"b": "PROVIDER_B_API_KEY"
}
}'
{
"data": {
"access_id": "1733e97b-a38a-43b9-8a85-61804f9ad93a",
"id": "949df7a9-8aaf-40d3-9efc-bd73d9dff075",
"model": "medium",
"prompt": "Hello!",
"provider": "cohere",
"raw": {
"generations": [...],
"id": "d8e53599-1d80-4ac1-a5d5-40450eedbd22",
"meta": {...},
"prompt":"Hello!"
},
"results": {
"generations": [" My name is Xiaomi, and I'm the leader of the algae from space, they live"]
},
"type": "generation"
}
}
Return a list of experiments for the given task
curl -XPOST https://promptmux.ai/api/v1/experiment/list \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project": "my_project",
"task": "my_task"
}'
{
"data": [
{
"ends_at": "2023-02-21T23:59:00Z",
"experiment": {
"prompt_a": "de53664b-20f8-4954-8159-61575009b063",
"prompt_a_fraction":50,
"prompt_b": "a953d329-8c32-48e8-8324-c5f7e6e16e94",
"prompt_b_fraction": 50
},
"id": "c72f7634-b381-46fb-8ea4-3b2ac1799e2c",
"type": "ab_test"
}
]
}