Artem Aliev

<img alt="bring your own spark" data-align="center" data-entity-type="file" data-entity-uuid="40ca7c7a-9d27-4bf7-abb4-508e68f2d954" height="336" src="https://www.datastax.com/sites/default/files/inline-images/fig1.png" width="758" />
<p><a href="https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/spark/byosIntro.html">Bring Your Own Spark</a>&nbsp;(BYOS) is a feature of DSE Analytics designed to connect from external Apache Spark™ systems to DataStax Enterprise with minimal configuration efforts. In this post we introduce how to configure BYOS and show some common use cases.</p>

<p>BYOS extends the&nbsp;<a href="https://github.com/datastax/spark-cassandra-connector/">DataStax Spark Cassandra Connector</a>&nbsp;with DSE security features such as Kerberos and SSL authentication. It also includes drivers to access the DSE Cassandra File System (CFS) and DSE File System (DSEFS) in 5.1.</p>

<p>There are three parts of the deployment:</p>

<ul>
	<li>&lt;dse_home&gt;clients/dse-byos_2.10-5.0.6.jar is a fat jar. It includes everything you need to connect the DSE cluster: Spark Cassandra Connector with dependencies, DSE security connection implementation, and CFS driver.</li>
	<li>'dse client-tool configuration byos-export' tool help to configure external Spark cluster to connect to the DSE</li>
	<li>'dse client-tool spark sql-schema' tool generates SparkSQL-compatible scripts to create external tables for all or part of DSE tables in SparkSQL metastore.</li>
</ul>

<p>HDP 2.3+ and CDH 5.3+ are the only Hadoop distributions which support Java 8 officially and which have been tested with BYOS in DSE 5.0 and 5.1.</p>

<h1>Quick Start Guide</h1>

<h2>Pre-requisites:</h2>

<p>There is installed and configured a Hadoop or standalone Spark system and you have access to at least one host on the cluster with a preconfigured Spark client. Let’s call it spark-host. The Spark installation should be pointed to by $SPARK_HOME.</p>

<p>There is installed and configured a DSE cluster and you have access to it. Let’s call it dse-host. I will assume you have a cassandra_keyspace.exampletable C* table created on it.The DSE is located at $DSE_HOME.</p>

<p>DSE supports Java 8 only. Make sure your Hadoop, Yarn and Spark use Java 8. See your Hadoop distro documentation on how to upgrade Java version (<a href="http://www.cloudera.com/documentation/enterprise/5-4-x/topics/cm_ig_java_home_location.html?scroll=cmig_topic_16">CDH</a>,&nbsp;<a href="http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_ambari_reference_guide/content/ch_changing_the_jdk_version_on_an_existing_cluster.html">HDP</a>).</p>

<h2>Prepare the configuration file</h2>

<p>On dse-host run:</p>

<pre>
$DSE_HOME/bin/dse client-tool configuration byos-export byos.conf</pre>

<p>It will store DSE client connection configuration in Spark-compatible format into byos.conf.</p>

<p>Note: if SSL or password authentication is enabled, additional parameters needed to be stored. See dse client-tool documentation for details.</p>

<p>Copy the byos.conf to spark-host.</p>

<p>On spark-host append the ~/byos.conf file to the Spark default configuration</p>

<pre>
cat byos.conf &gt;&gt; $SPARK_HOME/conf/conf/spark-defaults.conf</pre>

<p>Note: If you expect conflicts with spark-defaults.conf, the byos-export tool can merge properties itself; refer to the documentation for details.</p>

<h2>Prepare C* to SparkSQL mapping (optional)</h2>

<p>On dse-host run:</p>

<pre>
dse client-tool spark sql-schema -all &gt; cassandra_maping.sql</pre>

<p>That will create cassandra_maping.sql with spark-sql compatible create table statements.</p>

<p>Copy the file to spark-host.</p>

<h2>Run Spark</h2>

<p>Copy $DSE_HOME/dse/clients/dse-byos-5.0.0-all.jar to the spark-host</p>

<p>Run Spark with the jar.</p>

<pre>
$SPARK_HOME/bin/spark-shell --jars dse-byos-5.0.0-all.jar
scala&gt; import com.datastax.spark.connector._
scala&gt; sc.cassandraTable(“cassandra_keyspace”, "exampletable" ).collect</pre>

<p>Note: External Spark can not connect to DSE Spark master and submit jobs. Thus you can not point it to DSE Spark master.</p>

<h2>SparkSQL</h2>

<p>BYOS does not support the legacy Cassandra-to-Hive table mapping format. The spark data frame external table format should be used for mapping:&nbsp;<a href="https://github.com/datastax/spark-cassandra-connector/blob/master/doc/14_data_frames.md">https://github.com/datastax/spark-cassandra-connector/blob/master/doc/14_data_frames.md</a></p>

<p>DSE provides a tool to auto generate the mapping for external spark metastore: dse client-tool spark sql-schema</p>

<p>On the dse-host run:</p>

<pre>
dse client-tool spark sql-schema -all &gt; cassandra_maping.sql</pre>

<p>That will create cassandra_maping.sql with spark-sql compatible create table statements</p>

<p>Copy the file to spark-host</p>

<p>Create C* tables mapping in spark meta-store</p>

<pre>
$SPARK_HOME/bin/spark-sql--jars dse-byos-5.0.0-all.jar -f cassandra_maping.sql</pre>

<p>Tables are now ready to use in both SparkSQL and Spark shell.</p>

<pre>
$SPARK_HOME/bin/spark-sql --jars dse-byos-5.0.0-all.jar
spark-sql&gt; select * from cassandra_keyspace.exampletable</pre>

<p>&nbsp;</p>

<pre>
$SPARK_HOME/bin/spark-shell —jars dse-byos-5.0.0-all.jar
scala&gt;sqlConext.sql(“select * from cassandra_keyspace.exampletable");</pre>

<h2>Access external HDFS from dse spark</h2>

<p>DSE is built with Hadoop 2.7.1 libraries. So it is able to access any Hadoop 2.x HDFS file system.</p>

<p>To get access you need just proved full path to the file in Spark commands:</p>

<pre>
scala&gt; sc.textFile("hdfs://&lt;namenode_host&gt;/&lt;path to the file&gt;")</pre>

<p>To get a namenode host you can run the following command on the Hadoop cluster:</p>

<pre>
hdfs getconf -namenodes</pre>

<p>If the Hadoop cluster has custom configuration or enabled kerberos security, the configuration should be copied into the DSE Hadoop config directory:</p>

<pre>
cp /etc/hadoop/conf/hdfs-site.xml $DSE_HOME/resources/hadoop2-client/conf/hdfs-site.xml</pre>

<p>Make sure that firewall does not block the following HDFS data node and name node ports:</p>

<table>
	<tbody>
		<tr>
			<td>NameNode metadata service</td>
			<td>8020/9000</td>
		</tr>
		<tr>
			<td>DataNode</td>
			<td>50010,50020</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

<h2>Security configuration</h2>

<h3><b>SSL</b></h3>

<p>Start with truststore generation with DSE nodes certificates. If client certificate authentication is enabled (require_client_auth=true), client keystore will be needed.</p>

<p>More info on certificate generation:</p>

<p><a href="https://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html">https://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html</a></p>

<p>Copy both file to each Spark node on the same location. The Spark '--files' parameter can be used for the coping in Yarn cluster.</p>

<p>Use byos-export parameters to add store locations, type and passwords into byos.conf.</p>

<pre>
dse client-tool configuration byos-export --set-truststore-path .truststore --set-truststore-password 
password --set-keystore-path .keystore --set-keystore-password password byos.conf</pre>

<p>Yarn example:</p>

<pre>
spark-shell --jars byos.jar --properties-file byos.conf --files .truststore,.keystore</pre>

<h3><b>Kerberos</b></h3>
<img alt="kerberos" data-align="center" data-entity-type="file" data-entity-uuid="6a9b504e-1ed0-4020-bf01-a6d2276bea69" height="493" src="https://www.datastax.com/sites/default/files/inline-images/fig2.png" width="526" />
<p>Make sure your Spark client host (where spark driver will be running) has kerberos configured and C* nodes DNS entries are configured properly. See more details in the&nbsp;<a href="https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/sec/secSetKerbEnv.html">Spark Kerberos documentation</a>.</p>

<p>If the Spark cluster mode deployment will be used or no Kerberos configured on the spark client host use "Token based authentication" to access Kerberized DSE cluster.</p>

<p>byos.conf file will contains all necessary Kerberos principal and service names exported from the DSE.</p>

<p>The JAAS configuration file with the following options need to be copied from DSE node or created manually on the Spark client node only and stored at $HOME/.java.login.config file.</p>

<pre>
DseClient {
       com.sun.security.auth.module.Krb5LoginModule required
       useTicketCache=true
       renewTGT=true;
};</pre>

<p>Note: If a custom file location is used, Spark driver property need to be set pointing to the location of the file.</p>

<pre>
--conf 'spark.driver.extraJavaOptions=-Djava.security.auth.login.config=login_config_file'</pre>

<p>BYOS authenticated by Kerberos and request C* token for executors authentication. The token authentication should be enabled in DSE. the spark driver will automatically cancel the token on exit</p>

<p>Note: the CFS root should be passed to the Spark to request token with:</p>

<pre>
--conf spark.yarn.access.namenodes=cfs://dse_host/</pre>

<h3>Spark Thrift Server with Kerberos</h3>

<p>It is possible to authenticate services with keytab. Hadoop/YARN services already preconfigured with keytab files and kerberos useк if kerberos was enabled in the hadoop. So you need to grand permissions to these users. Here is example for hive user</p>

<pre>
cqlsh&gt; create role 'hive/hdp0.dc.datastax.com@DC.DATASTAX.COM' with LOGIN = true;</pre>

<p>Now you can login as a hive kerberos user, merge configs and start Spark thrift server. It will be able to query DSE data:</p>

<pre>
#&gt; kinit -kt /etc/security/keytabs/hive.service.keytab \ hive/hdp0.dc.datastax.com@DC.DATASTAX.COM
#&gt; cat /etc/spark/conf/spark-thrift-sparkconf.conf byos.conf &gt; byos-thrift.conf
#&gt; start-thriftserver.sh --properties-file byos-thrift.conf --jars dse-byos*.jar</pre>

<p>Connect to it with beeline for testing:</p>

<pre>
#&gt; kinit
#&gt; beeline -u 'jdbc:hive2://hdp0:10015/default;principal=hive/_HOST@DC.DATASTAX.COM'</pre>

<h3>Token based authentication</h3>

<p>Note: This approach is less secure than Kerberos one, use it only in case kerberos is not enabled on your spark cluster.</p>

<p>DSE clients use hadoop like token based authentication when Kerberos is enabled in DSE server.</p>

<p>The Spark driver authenticates to DSE server with Kerberos credentials, requests a special token, send the token to the executors. Executors authenticates to DSE server with the token. So no kerberos libraries needed on executors node.</p>

<p>If the Spark driver node has no Kerberos configured or spark application should be run in cluster mode. The token could be requested during configuration file generation with --generate-token parameters.</p>

<pre>
$DSE_HOME/bin/dse client-tool configuration byos-export --generate-token byos.conf</pre>

<p>Following property will be added to the byos.conf:</p>

<pre>
spark.hadoop.cassandra.auth.token=NwAJY2Fzc2FuZHJhCWNhc3NhbmRyYQljYXNzYW5kcmGKAVPlcaJsigFUCX4mbIQ7YU_yjEJgRUwQNIzpkl7yQ4inoxtZtLDHQBpDQVNTQU5EUkFfREVMRUdBVElPTl9UT0tFTgA</pre>

<p>It is important to manually cancel it after task is finished to prevent re usage attack.</p>

<pre>
dse client-tool cassandra cancel-token NwAJY2Fzc2FuZHJhCWNhc3NhbmRyYQljYXNzYW5kcmGKAVPlcaJsigFUCX4mbIQ7YU_yjEJgRUwQNIzpkl7yQ4inoxtZtLDHQBpDQVNTQU5EUkFfREVMRUdBVElPTl9UT0tFTgA</pre>

<h2>Instead of Conclusion</h2>

<p>Open Source Spark Cassandra Connector and Bring Your Own Spark feature comparison:</p>

<table>
	<tbody>
		<tr>
			<td><b>Feature</b></td>
			<td><b>OSS</b></td>
			<td><b>DSE BYOS</b></td>
		</tr>
		<tr>
			<td>DataStax Official Support</td>
			<td>No</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>Spark SQL Source Tables / Cassandra DataFrames</td>
			<td>Yes</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>CassandraDD batch and streaming</td>
			<td>Yes</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>C* to Spark SQL table mapping generator</td>
			<td>No</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>Spark Configuration Generator</td>
			<td>No</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>Cassandra File System Access</td>
			<td>No</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>SSL Encryption</td>
			<td>Yes</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>User/password authentication</td>
			<td>Yes</td>
			<td>Yes</td>
		</tr>
		<tr>
			<td>Kerberos authentication</td>
			<td>No</td>
			<td>Yes</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>


Bring Your Own Spark

Artem Aliev

Discover more

Share

More Technology

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Artem Aliev

Discover more

Share

More Technology

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale