Russell Spitzer

<h2>Introduction</h2>

<p>DSE Search (Apache Solr based) and DSE Analytics (Apache Spark Based) may seem like they are basically designed for orthogonal use cases. Search optimizes quick generic searches over your Big Data and Analytics optimizes for reading your entire dataset for processing. &nbsp;But there is a sweet spot where Analytics can benefit greatly from the enhanced indexing capabilities from Search. Previously in DSE this synergy could only be accessed from the RDD API but now with DSE 5.1 we bring DSE Search together with DSE Analytics in SparkSQL and DataFrames.</p>

<h2>Requirements</h2>

<p>DSE Search must be enabled in the target datacenter of the SparkSql or Dataframe request. DSE Analytics must be enabled on host datacenter of the request. Locality is only guaranteed if both Search and Analytics are colocated in the same Datacenter.</p>

<h2>Turning on and off the optimizations</h2>

<p>In 5.1 this feature is Opt-In and is only useful in certain scenarios which we currently do not automatically detect. To enable these optimizations set the&nbsp;spark.sql.dse.solr.enable_optimization=true&nbsp;as a Spark configuration option. This can be set at the application level by setting the value in</p>

<ul>
	<li>spark-defaults</li>
	<li>on the command line with --conf</li>
	<li>programatically in the Spark Conf</li>
</ul>

<p>It can also be set per DataFrame by passing the parameter as an option to your DataFrame</p>

<p>import org.apache.spark.sql.cassandra._<br />
spark.read<br />
  .cassandraFormat("ks", "tab")<br />
  .option("spark.sql.dse.solr.enable_optimization", "true")</p>

<h2>How it Works</h2>

<p>When a query is sent to Spark via SparkSql or Dataframes it ends up in an optimization engine called [Catalyst](<a href="https://spark-summit.org/2016/events/deep-dive-into-catalyst-apache-spark-20s-optimizer/">https://spark-summit.org/2016/events/deep-dive-into-catalyst-apache-spark-20s-optimizer/</a>).This engine reduces the query into a set of standardized operations and predicates. Some of those predicates are presented to the data source (which can be Cassandra). The Datasource is then able to decide whether or not it can handle the predicates presented to it.</p>

<p>When Solr Optimization is enabled DSE adds a special predicate handling class to the Cassandra DataSource provided in the Datastax Spark Cassandra Connector, &nbsp;allowing DSE to transform the Catalyst Predicates into Solr Query clauses.</p>
<img alt="Solr optimization" data-align="center" data-entity-type="file" data-entity-uuid="cb2a2e0a-2cab-46e4-bce5-f654ec8b9673" src="https://www.datastax.com/sites/default/files/inline-images/Solr-Optimization.png" />
<p>These compiled Solr clauses are then added to the full table scan done by CassandraTableScanRDD. Once you have enabled Solr optimization these transformations will be done whenever applicable. If a predicate cannot be handled by Solr we push it back up into Spark.</p>

<h2>Debugging / Checking whether predicates are applied</h2>

<p>The optimizations are all done on the Spark Driver (the JVM running your job). To see optimizations as they are being planned add the following line to the logback-spark file which is used for your driver</p>

<p>&lt;logger name="org.apache.spark.sql.cassandra.SolrPredicateRules" level="DEBUG"/&gt;<br />
<br />
logback-spark-shell.xml # dse spark<br />
logback-spark-sql.xml # dse spark-sql<br />
logback-spark.xml # dse spark-submit</p>

<p>This will log the particular operations performed by the DSE added predicate strategy. If you would like to see all of the predicate strategies being applied add</p>

<p>&lt;logger name="org.apache.spark.sql.cassandra" level="DEBUG"/&gt;</p>

<p>as well.</p>

<h2><strong>Performance Details</strong></h2>

<h2>Count Performance</h2>

<p>The biggest performance difference for using Solr is for&nbsp;count&nbsp;style queries where every predicate can be handled by DSE Search. In these cases around 100X the performance of the full table scan can be achieved.<br />
This means analytics queries like "SELECT COUNT(*) where Column &gt; 5" can be done in near-real time by automatically routing through DSE Search. (CP in this graph refers to Continuous Paging another DSE Exclusive Analytics Feature)</p>
<img alt="Solr counts vs spark counts bar chart" data-align="center" data-entity-type="file" data-entity-uuid="0cf22ebd-bd47-41fb-9cbe-ce7cdc92c938" src="https://www.datastax.com/sites/default/files/inline-images/chart_0.png" />
<h2>Filtering Performance</h2>

<p>The other major use case has to do with filtering result sets. This comes into play when retrieving a small portion of the total dataset. Search uses point partition key lookups when retrieving records so there is a linear relationship between the number of rows being retrieved and the time it takes to run. Normally Analytics performs full table scans, whose runtimes are independent of the amount the data actually being filtered in Spark. The speed boosts to DSE Analytics with Continuous Paging have set a low tradeoff point between Search doing specific lookups and running a full scan in Spark. This means that unless the dataset is being filtered down to a few percent of the total data size it is better to leave Search Optimization off.</p>
<img alt="Retrieve small portions of data fast barchart" data-align="center" data-entity-type="file" data-entity-uuid="8625a876-3542-43b3-8b84-26b5895790b2" src="https://www.datastax.com/sites/default/files/inline-images/chart-1.png" />
<p>In the above chart, the yellow line (Spark Only) runs at a constant time since it always requires reading the same amount of data (all of it). The Blue and Red lines show that since Search is individually requesting rows, it's duration is dependent on the amount of records retrieved. Since Primary Keys and Normal Columns are stored differently the performance of retrieving primary keys only (Red) is faster than retrieving the whole row (Blue).</p>

<p>For retrieving whole rows (blue), the inflection point is around 1.5 percent of the total data size. If the filter returns less rows than this percentage performance will be better using the Search optimization.</p>

<p>For returning only primary key values (red), the inflection point is around 5 percent of the total data size. Any filters returning less than this percentage will perform better using the Search optimization.</p>

<p>These are some general guidelines and the performance of any system may be different depending on data layout and hardware. .</p>

<h2>Caveats</h2>

<p>One major caveat to note is that currently the&nbsp;SELECT COUNT(*)&nbsp;query without any predicates currently will only trigger a normal Cassandra pushdown count. This can be forced to use a solr count instead by adding a "IS NOT NULL" filter on a partition key column. We plan on including this as an automatic optimization in the future.</p>

<p>All results returned via DSE Search are subject to the same limitations that DSE Search is bound by. &nbsp;The accuracy of the counts are dependent on the consistency of the Solr core on the replicas that provide the results. Errors at indexing time may persist until manually corrected, especially since the Spark Connector doesn't actually use the same endpoint selection logic DSE Search does.</p>


DSE 5.1: Automatic Optimization of Spark SQL Queries Using DSE Search

Russell Spitzer

Discover more

Share

More Company

DataStax Astra DB Nabs Three Prestigious 2023 TrustRadius “Best of” Awards, Dominates the Vector Databases Category

DataStax Has Achieved ISO 27001 Certification

Keeping the Momentum Rolling with Google Cloud

My Internship Journey: A Dive into AI

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Russell Spitzer

Discover more

Share

More Company

DataStax Astra DB Nabs Three Prestigious 2023 TrustRadius “Best of” Awards, Dominates the Vector Databases Category

DataStax Has Achieved ISO 27001 Certification

Keeping the Momentum Rolling with Google Cloud

My Internship Journey: A Dive into AI

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale