Sebastian Estevez

<section>This post is an intro to DSE Graph with a focus on the strategies that should be used to load large graphs with billions of vertices and edges. For those familiar with DSE Graph and large graph strategies or those who want to dive directly into loading data, proceed to the next post in this two part series entitiled Large Graph Loading Best Practices: Tactics
<p>&nbsp;</p>

<h3 id="introtodsegraph">Intro to DSE Graph</h3>

<p>DSE Graph is differentiated from other graph databases by building on DataStax Enterprise's scalability, replication, and fault tolerance.</p>

<p><strong><em>Note</em></strong><em>&nbsp;- To understand how DSE Graph data is stored in DSE's Apache Cassandra(TM) storage engine, check out&nbsp;<a href="https://www.datastax.com/dev/blog/a-letter-regarding-native-graph-databases">Matthias</a> Marko's posts on the matter.</em></p>

<p>When folks ask me where DSE Graph falls in the greater database / graph database landscape, I use this image to communicate the combination of scalability and value in relationships that make DSE Graph such a unique product:</p>

<p>DSE Graph is positioned on the right side of the chart where relationships are most valuable, and toward the top of the chart due to the scalability it inherits from DSE and Cassandra. The third key aspect that differentiates DSE graph is the velocity of the data it can support. Unlike analytical graph engines which load a static graph to memory and then crunch that static graph for insights, an operational graph is constantly changing as the real world concepts whose relationships and vertices it represents are created, updated, and deleted.</p>

<p><strong><em>Key Takeaway</em></strong><em>&nbsp;- DSE Graph is designed as a real-time, operational, distributed graph database.</em></p>

<h3 id="motivationandgoalsplayingwithscalablegraphs">Motivation and Goals: Playing with Scalable Graphs</h3>

<p>If you have distributed graph problem, you may want to bulk load your data into DSE graph and start querying. However, loading significant amounts of data (&gt;1 billion V's or E's) into graph dbs is a time consuming, nontrivial task. The purpose of this article is to summarize some key design considerations related to dealing with large graphs.</p>

<h3 id="largegraphsidempotenceandscalability">Large graphs, idempotence, and scalability</h3>

<p>Idempotence is a common concept in distributed systems design. If an operation is idempotent, it can be repeated over and over and still yield the same result. We use idempotence to help us solve problems like the fact that&nbsp;<a href="http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/">exactly once delivery does not exist</a>, it also greatly simplifies the design of our systems, minimizing bugs and promoting maintainablility. For the purposes of two part article series, we are going to focus on building scalable distributed idempotent graphs. This is one of the design choices that is supported in DSE Graph but note that not all graphs that can be built on DSE graph will have idempotent vertices and idempotent edges.</p>

<h4 id="idempotentvertices">Idempotent Vertices</h4>

<p>DSE graph allows two types of vertices, 1) those with system generated keys and 2) those with custom ids. For the purposes of this two part series we are going to concentrate on custom ids. Custom ids are useful for large graph problems in that they allow developers to take graph partitioning into their own hands. Custom ids will feel familiar if you have used DSE or cassandra and understand data modeling.</p>

<p>You configure the partition key of your Vertex label with a DDL operation:</p>

<pre>
<code>schema().vertexLabel('MachineSensor').partitionKey('manufacturing_plant_id').clusteringKey('sensor_id').create()
</code></pre>

<p>If you are using custom ids, the partition key is required and the clustering key is optional. For more on Cassandra data modeling and clustering keys vs. partition keys see my post on&nbsp;<a href="http://www.sestevez.com/large-graph-strategies-part-1/www.sestevez.com/data-modeler/">data modeling for DSE</a>.</p>

<p><strong><em>Note</em></strong>&nbsp;<em>- your partition key, clustering key combination should provide uniqueness for the vertex. With this configuration, reinserting will not generate duplicates</em></p>

<h4 id="idempotentedges">Idempotent Edges</h4>

<p>DSE Graph edges support different cardinality options. For multiple cardinality edges (where there can be more than one edge between the same two vertices of the same edge label type) edge creation is&nbsp;<strong>not</strong>&nbsp;idempotent.</p>

<p>For the purposes of this two part article series, we will focus on single cardinality (thereby idempotent) edges. You can create single cardinality edge lables in DSE Graph use the&nbsp;<code>single()</code>&nbsp;keyword:</p>

<pre>
<code>schema.edgeLabel('has_sensor').single().create()
</code></pre>

<h3 id="letsload">Let's load!</h3>
<img alt="DSE Graph Landscape" data-align="center" data-entity-type="file" data-entity-uuid="cf58dfef-67ab-4d74-8f9e-acc8e6bea43a" src="https://www.datastax.com/sites/default/files/inline-images/graph_landscape.png" /></section>

<footer>
<section>
<h4><a href="http://www.sestevez.com/author/sebastian/">Sebastian Estevez</a></h4>

<p>#Datastax #Cassandra I like music, technology, and building things @syllogistic on twitter, statements are my own</p>

<p>NYC&nbsp;<a href="http://sestevez.com/">http://sestevez.com</a></p>
</section>
</footer>


Large Graph Loading Best Practices: Strategies (Part 1)

Sebastian Estevez

Discover more

Share

More Technology

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

DataStax and AWS: Accelerating GenAI Applications into Production

Turn Product Docs into a RAG-Powered App with Vercel and Astra DB

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Sebastian Estevez

Discover more

Share

More Technology

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

DataStax and AWS: Accelerating GenAI Applications into Production

Turn Product Docs into a RAG-Powered App with Vercel and Astra DB

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale