Alan Ho

Abhilash Sharma

Building generative AI prototypes with retrieval augmented generation (<a href="https://www.datastax.com/guides/what-is-retrieval-augmented-generation">RAG</a>) is relatively easy, but what does it take to get an application to production? <a href="https://skypoint.ai/">Skypoint</a>, a provider of AI-based solutions for the senior care industry, has notched some significant successes, using <a href="https://www.datastax.com/guides/what-is-a-large-language-model">large language models</a> to improve data accessibility, provide conversational AI, and streamline administrative processes at senior care facilities.&nbsp;
In a recent webinar hosted by DataStax, SkyPoint founder and CEO Tisson Mathew and Skypoint principal engineer Abhilash Sharma shared tips and insights to consider when building generative AI applications.
<iframe src="https://www.youtube.com/embed/_3e3Hk31h7k?si=kloEr5LVIV3_-e0b" width="560" height="314" allowfullscreen="allowfullscreen"></iframe>
<h2>Use case</h2>
Skypoint&rsquo;s generative AI application gives healthcare providers instant access to public and private data. Skypoint offers both its own interface for users as well as a <a href="https://skypoint.ai/skypoint-ai-chatgpt-plugin/">ChatGPT interface</a>:&nbsp;
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://cdn.sanity.io/images/bbnkhnhl/production/d53f63d9cfb84488440b6f3de4061013c56de079-512x394.png" alt="" width="550" height="423" />
Mathew detailed some examples of AI interactions, along with how they relate to SkyPoint&rsquo;s goal of improving patient care:
&ldquo;Do they (patients) have care gaps? How do you actually take care of them? What type of clinical protocols are there? So eventually as we are building this, we want the AI agents to write the care plan for a patient. Essentially you are looking at PHI (personal health information) data, compliance, clinical protocols, all of that. And that's how you write a care plan.&rdquo;
<h2>Architecture</h2>
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://cdn.sanity.io/images/bbnkhnhl/production/aa251ad7254157df180ee1e91d2b526b1cd6a171-512x288.png" alt="" width="600" height="337" />
The Skypoint AI application is driven by a combination of structured and unstructured data. Structured data (elderly care information and assisted living ratings, for example) is stored in its <a href="https://skypoint.ai/platform/lakehouse/">lakehouse</a> (backed by Databricks). Unstructured data (such as policy PDFs and legal documents) are stored in <a href="https://www.datastax.com/products/vector-search">DataStax Astra DB with vector search</a>.
LLMs can handle structured and unstructured data, making them versatile for various healthcare applications, such as extracting information from electronic health record (EHR) systems, analyzing vital signs, and answering patient-level questions.
Skypoint AI uses several cutting-edge techniques such as <a href="https://arxiv.org/abs/2210.03629">ReAct</a> to use LLMs to determine what data should be queried for a given prompt. If the data is unstructured, it performs the <a href="https://www.datastax.com/products/vector-search">traditional RAG workflow</a>.
&nbsp;
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://cdn.sanity.io/images/bbnkhnhl/production/5625e919d7e3bb8523c3024275b6cd5302efd644-512x288.png" alt="" width="600" height="337" />
For structured data queries, Skypoint AI translates the natural language query into SQL and executes it against its lakehouse.
&nbsp;
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://cdn.sanity.io/images/bbnkhnhl/production/6eca6e8dbd61341b686b1a26fd6245d6dce62867-512x288.png" alt="" width="600" height="337" />
<h2>Vector database use case: Multiquery RAG&nbsp;</h2>
When performing unstructured queries, a single query is turned into multiple queries to increase the amount of data that is summarized and returned to users. This is similar to forward-looking active retrieval (<a href="https://arxiv.org/abs/2305.06983">paper</a>, <a href="https://cassio.org/demo_apps/flare_langchain/">code</a>), and advanced RAG techniques to improve the answer quality and reduce hallucinations.&nbsp;
Skypoint AI uses Apache Cassandra&rsquo;s intrinsic ability to parallelize queries to achieve fast performance for multi-query RAG within the strict latency SLAs.
<h2>Vector database use case: Few-shot prompting</h2>
In the scenario of generating SQL statements from natural-language queries, the quality of the SQL can be greatly enhanced using few-shot prompting. Example natural language -&gt; SQL pairs are stored in the <a href="https://www.datastax.com/guides/what-is-a-vector-database">vector database</a>. Upon getting a new natural language query, multiple similar natural queries and SQL pairs are put into the prompt (this is also known as chain-of-thought), and SQL is generated. Few-shot prompting is especially useful when generating&nbsp; SQL statements with JOINS.
<h2>Vector DB Use Case: LLM Caching</h2>
When performing structured queries, Skypoint needs serial calls to LLMs and databases to retrieve schemas and interpret them to generate the appropriate SQL statement for querying the database. This can result in an unacceptable delay in responding to the user. By using the vector store as an LLM cache, the multiple LLM calls can be eliminated, drastically improving the performance of the database.
Skypoint uses Cassandra&rsquo;s intrinsic fast response times and ability to index new data in its vector store immediately.
<h2>Vector DB Use Case: Store conversation traces for debugging and fine-tuning</h2>
Storing the thought traces for future analysis serves three purposes:
<ol>
<li style="font-weight: 400;" aria-level="1">Adding conversational context In a conversation with an LLM, properly selecting components of a previous conversation helps the LLM give more contextually relevant answers.</li>
<li style="font-weight: 400;" aria-level="1">Debugging By capturing conversations, one can analyze offline whether Skypoint&rsquo;s agent is correctly answering the customer&rsquo;s inquiry</li>
<li style="font-weight: 400;" aria-level="1">Fine-tuning After accumulating enough thought traces, you can fine-tune the model with the thought traces to ensure that the steps taken to answer a question are more deterministic.</li>
</ol>
Skypoint uses DataStax Astra DB&rsquo;s ability to cost-efficiently store, query, and export this information for the purposes of offline analysis or fine-tuning.
<h2>Technical and business tips for Gen AI</h2>
Add a validation layer (<a href="https://youtu.be/_3e3Hk31h7k?feature=shared&amp;t=2477">skip to webcast segment</a>) - To reduce hallucinations when performing structured queries, Skypoint performs a validation using an LLM to ensure that the answer is consistent with the question being asked.
Focus on real-world use cases (<a href="https://youtu.be/_3e3Hk31h7k?feature=shared&amp;t=2827">skip to webcast segment</a>) - In healthcare, LLMs can answer questions, reduce administrative burdens, and improve patient care. Real-world applications include question-answering, creating care plans, and providing data-driven insights.
Structured and unstructured data integration (<a href="https://youtu.be/_3e3Hk31h7k?feature=shared&amp;t=2827">skip to webcast segment</a>) - LLMs can handle both structured and unstructured data, making them versatile for various healthcare applications, such as extracting information from EHR systems, analyzing vital signs, and answering patient-level questions.
Progressive implementation (<a href="https://youtu.be/_3e3Hk31h7k?feature=shared&amp;t=2827">skip to webcast segment</a>) - When rolling out generative AI applications, follow a step-by-step process, starting with a subscription to the platform, then a proof of concept (POC) or MVP, and gradually expand to different use cases. Ensure security, privacy, and compliance at each stage.
Educate and train users (<a href="https://youtu.be/_3e3Hk31h7k?feature=shared&amp;t=2827">skip to webcast segment</a>) - Training and educating users on interacting with the generative AI system is crucial. Feedback and user adoption are essential for the success of AI applications.
Plan for the future (<a href="https://youtu.be/_3e3Hk31h7k?feature=shared&amp;t=2827">skip to webcast segment</a>) - Consider the future capabilities of <a title="What is Generative AI" href="https://www.datastax.com/guides/what-is-generative-ai">generative AI</a>, such as autonomous agents, and plan for advanced use cases beyond the initial implementation.
<h2>Conclusion</h2>
RAG application prototypes are relatively simple to build, but creating a production-grade application requires highly sophisticated techniques to reduce hallucinations and take advantage of both structured and unstructured data.&nbsp;
Skypoint uses a combination of advanced RAG techniques and the advanced capabilities of Cassandra/Astra DB to achieve production-grade customer experiences for regulated industries like healthcare. If you want to replicate some of the techniques outlined here, <a href="https://www.datastax.com/products/datastax-astra">sign up to try Astra DB</a>.
For more about Skypoint, check out&nbsp; &ldquo;<a href="https://www.youtube.com/watch?v=nUrpVDJFCng">Path to Production: Skypoint&rsquo;s Journey with LangChain and Apache Cassandra</a>,&rdquo; with <a href="https://www.linkedin.com/in/ACoAAB06lJkB2-hiC0mqJ2yHgFtnAbf4UFXQ4Dg">Harrison Chase</a>, CEO of <a href="https://www.linkedin.com/company/langchain/">LangChain</a>, and <a href="https://www.linkedin.com/in/ACoAAAADIVABLA5dh4fVM4Htebq8B9vIgOlhbBQ">Tisson Mathew</a>, CEO of <a href="https://www.linkedin.com/company/skypoint-ai/">Skypoint</a>, as they share experiences and insights in taking LLM apps into production via on-demand here.
<h2>Technical References</h2>
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.google.com/presentation/d/1l4EuF4xUM5XpzRAbndjPnElgi2JxK2fZ/edit#slide=id.p1">SkyPoint Architecture Diagrams</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://arxiv.org/pdf/2201.11903.pdf">Chain of Thought</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://arxiv.org/abs/2306.00739">Text-to-SQL</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://arxiv.org/pdf/2210.11416.pdf">Instruction Fine-Tuning</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://arxiv.org/abs/2210.03629">ReAct</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://arxiv.org/abs/2305.06983">FLARE Paper</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://cassio.org/demo_apps/flare_langchain/">FLARE Implementation with LangChain and Astra DB</a></li>
</ul>

SkyPoint: How to Train Your LLM Dragon

Alan HoVP Product, AI

Abhilash SharmaPrincipal data engineer, SkyPoint

Discover more

Share

Share

Use case

Architecture

Vector database use case: Multiquery RAG

Vector database use case: Few-shot prompting

Vector DB Use Case: LLM Caching

Vector DB Use Case: Store conversation traces for debugging and fine-tuning

Technical and business tips for Gen AI

Conclusion

Technical References

More Success Stories

How PTC Is Transforming eCommerce with Generative AI, DataStax Astra DB, and Vector Search

LangChain and DataStax Astra DB Power Esynergy’s Retrieve-Augment Architecture for Sales Copilot

Vector Search in Action: How Restworld Improves Relevance and Accuracy in Job Matches

Restworld: Revolutionizing Hospitality Recruitment with Astra DB and Vector Search

One-stop Data API for Production GenAI