Sebastian Estevez

On November 8, OpenAI CEO Sam Altman took the stage to unveil the OpenAI Assistants API beta preview. This kicked off an exciting and busy time at DataStax as we set out to build and improve an API-compatible service powered by <a title="DataStax Astra DB" href="https://www.datastax.com/products/datastax-astra">Astra DB</a> and its <a title="DataStax Vector Search" href="https://www.datastax.com/products/vector-search">vector search capabilities</a>.
We <a href="https://www.datastax.com/blog/introducing-the-astra-assistants-api">launched the service</a> eight days later; today, we&rsquo;re excited to announce that we&rsquo;re <a href="https://github.com/datastax/astra-assistants-api">open sourcing the codebase</a> for users who want to see exactly how it works and for those that want to host it themselves.
Here's what we've been up to since launch (we've been busy!):
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://cdn.sanity.io/images/bbnkhnhl/production/9d15d11ced3408086bcd26d6c549643349b0a72b-1999x782.png" alt="" width="650" height="254" />
<h2>Why open source?</h2>
We believe that making the server code open source will benefit users in a few ways:
<ol>
<li style="font-weight: 400;" aria-level="1">It enables folks who want to deploy on premises / self hosted and point to their own self managed cassandra / DSE databases or locally hosted <a title="What is a Large Language Model" href="https://www.datastax.com/guides/what-is-a-large-language-model">LLM</a> inference servers</li>
<li style="font-weight: 400;" aria-level="1">Astra Assistants API users may be interested in exactly how we implement retrieval and function calling. This is an area that we are quickly innovating in and are interested in accelerating user feedback.</li>
</ol>
<h2>Internals</h2>
The <code>assistant-api-server</code> repo is a Python server app that heavily depends on FastAPI, Pydantic, and the DataStax Python CQL driver. It relies on LiteLLM for third-party LLM support and we've been very happy with their responsiveness in Github as well as their ability to quickly add new models as the AI landscape evolves.
The app is mostly stateless (with the exception of a database connection cache) and all authentication tokens and LLM provider configuration are passed as http headers. The <code>streaming-assistants</code> Python library makes it easy for users to store these configurations as environment variables and it takes care of the rest. We serve the app in production using uvicorn (a high performance web server for python) and scale it in Kubernetes and horizontal pod autoscaler. The Docker image is available at <code><a href="https://hub.docker.com/r/datastax/astra-assistants">datastax/astra-assistants</a>.</code>
The app consists of both generated and hand-written code. The generated code is based on OpenAI's <a href="https://github.com/openai/openai-openapi">OpenAPI spec</a> and generated with openapi-generator-cli from openapi-generator.tech. It mostly lives in the <code>openapi_server</code>&nbsp;directory. Leveraging the OpenAPI spec was one of the first design decisions we made and it was a no brainer: OpenAI's high quality spec (they use it to generate their SDKs) ensures that all the types for all the endpoints are built correctly and enforced by Pydantic.
The hand-written code takes the method stubs from <code>open-api-server/apis</code>&nbsp;and implements them using the types from <code>openapi-server/models</code> inside of <code>impl/routes</code>. The third-party LLM support is abstracted in <code>impl/services/inference_utils.py</code> and the database interactions occur in <code>impl/astra_vector.py</code>. We collect throughput, duration, and payload size metrics and export them with a Prometheus exporter that&rsquo;s exposed with a <code>/metrics</code> endpoint. The Prometheus exporter is configured to work using a <a href="https://github.com/prometheus/client_python/blob/master/prometheus_client/multiprocess.py">multi-process collector</a> to support our multi-process uvicorn production deployment.
Finally, in the <code>tests</code>&nbsp;directory we have implemented tests and CI using both an HTTP client directly (originally generated by openapi-generator.tech and tweaked manually) and custom tests that use both the OpenAI SDK and our <code>streaming-assistants</code> library directly.
<h2>Try it out!</h2>
The service has been running in production since November 15th, 2023 and we're constantly improving it. <a href="https://docs.datastax.com/en/astra/astra-db-vector/api-reference/astra-assistants-api.html">Try the service</a>, check out <a href="https://github.com/datastax/astra-assistants-api/tree/main">our code</a>, and give us <a href="https://github.com/datastax/astra-assistants-api/issues">feedback</a> on GitHub. We can't wait to see what you build!

The Astra Assistants API Is Now Open Source

Sebastian EstevezDataStax

Discover more

Share

Share

Why open source?

Internals

Try it out!

More Technology

How to Build a Crystal Image Search App with Vector Search

Knowledge Graphs for RAG without a GraphDB

How Winweb Built its AI Assistant with DataStax Astra DB and LangChain

Vercel + Astra DB: Get Data into Your GenAI Apps Fast

One-stop Data API for Production GenAI