Amanda Moran

<p>A natural language processing example using DataStax Enterprise Analytics with Apache Cassandra andApache Spark, Python, Jupyter Notebooks, Twitter API, Pattern (python package), and Sentiment Analysis</p>

<p><img alt="all logos" data-entity-type="file" data-entity-uuid="793421e6-af04-40de-82f1-aadcc0d058f3" src="https://www.datastax.com/sites/default/files/inline-images/allLogos.png" /></p>

<p>&nbsp;</p>

<h2>What Problem Are We Trying to Solve?</h2>

<p>The question of our time: "What movie should I actually see?" Wouldn't it be great if you could ask 1 million people this question? Wouldn't it be great if I could automate this process?</p>

<p>Data analytics doesn't have to be complicated! It can be as simple as trying to figure out what movie to see!</p>

<p>To do this we can use the power of Big Data, and power of a combination of technologies: DataStax Enterprise Analytics with Apache Spark and Apache Cassandra, Spark Machine Learning Libraries, Python, Pyspark, Twitter Tweets, Twitter Developer API, Jupyter notebooks, Pandas, and a python package Pattern.</p>

<h2>How Are We Going to Solve It?</h2>

<p>The Jupyter notebook that is available on Github <a href="https://github.com/amandamoran/sentimentAnalysisDSE">https://github.com/amandamoran/sentimentAnalysisDSE</a> will walk us through each of these following steps to get to our ultimate answer "Should I see this movie?"</p>

<p><img alt="Jupyter Notebook: Should I see this movie?" data-entity-type="file" data-entity-uuid="a48f57d7-d5b2-4d60-94ab-0ea14dc887e2" src="https://www.datastax.com/sites/default/files/inline-images/Screen%20Shot%202018-09-21%20at%2012.42.05%20PM.png" /></p>

<h2>Why Did We Choose Apache Cassandra and Apache Spark?</h2>

<h3>Apache Cassandra</h3>

<p>Apache Cassandra is a highly available distributed decentralized database.</p>

<p>The 4 major advantages of Cassandra are:</p>

<ul>
	<li>Distribution</li>
	<li>Replication</li>
	<li>Elastically Scalable</li>
	<li>High Availability&nbsp;</li>
</ul>

<p><em>Distributed</em></p>

<p>Each node in the cluster does the exact same job and is available for reads and writes from the client. Because each node in the cluster is the same, adding and removing nodes is a simple process and requires no downtime – not even a restart.</p>

<p><em>Replication</em></p>

<p>A distributed database that is highly available, which means surviving a node going down, requires that data be copied to more than one node. Data is replicated out through the cluster asynchronously through peer-to-peer communication. Replication factor can be set by the user and can be anywhere from 1 replica of the data to the number of nodes you have in your cluster.</p>

<p><em>Elastically Scalable</em></p>

<p>The read and write path of Cassandra allows for very high performance. Cassandra is elastically scalable, as more nodes are added (which there is actually no set upper limit!) performance increases linearly. Both reads and writes scale linearly, as you increase the number of nodes the ability to read and write data is increased.&nbsp;</p>

<p><em>High Availability</em></p>

<p>Cassandra has no single-point of failure because of the lack of a master-worker architecture. The lack of a master node is what allows for such high levels of availability. Cassandra was built with the expectation that nodes in the cluster will fail and because of this when it does, our database doesn't panic.&nbsp;</p>

<p>Think about your application:</p>

<ul>
	<li>Do you have Big Data (a lot of it!)?</li>
	<li>Do you need to be able to read/write fast?</li>
	<li>Do you need to be able to scale up/down easily?</li>
	<li>Do you need High Availability?</li>
	<li>Do you need multiple data center support?</li>
	<li>Multi-cloud/hybrid cloud support?</li>
</ul>

<p>If you application needs any of these things, you want to consider Cassandra!</p>

<h3>Apache Spark</h3>

<p>“Apache Spark is a unified analytics engine for large-scale data processing” --<a href="https://spark.apache.org/">https://spark.apache.org</a>.</p>

<p>Spark is over 100 times faster than Hadoop for analytics, as it utilizes in-memory processing and amazing parallelism. Spark has a variety of Machine Learning Libraries to do text analytics, transformations, clustering, collaborative filtering, classification and more. Spark also has Spark SQL, Spark Streaming, and SparkR. Spark is for performing analytics at scale on very large datasets quickly.</p>

<p>Think about your application:</p>

<ul>
	<li>Do you have Big Data?</li>
	<li>Do you need High Availability?</li>
	<li>Do you need analytics at lightning speed?</li>
	<li>Do you need a simple way to get insights into your data?</li>
</ul>

<p>If you application needs any of these things, you want to consider Spark!</p>

<h3>DataStax Enterprise Analytics</h3>

<p>Spark integration comes out of the box in DataStax Enterprise Analytics. Cassandra can connect to a Spark instance with connectors, but this all must be configured and managed by hand. DSE Analytics has its own file system (DSEFS) that does not have a single point of failure, where the namenode is a single point of failure is HDFS which does not allow for high availability. DSE comes with a highly available Spark Resource Manager -- out of the box! There is no special setup required to enable HA for the Spark Resource Manger on DSE.</p>

<p>Also get all the other benefits of DSE over Cassandra, support, easy of deployment, more testing, a graph engine, search. The list keeps going!</p>

<h3>Sentiment Analysis</h3>

<p>What IS Sentiment Analysis? Sentiment Analysis at a high level is very simple, it is natural language processing and text analytics to determine if a word or sentence is:</p>

<ul>
	<li>Positive</li>
	<li>Negative</li>
	<li>Neutral</li>
</ul>

<p>This is very easy concept for humans to understand, but its very difficult to teach a machine how to do this. Actually, is it that easy for humans to understand and do text analytics? Who hasn't gotten that confusing text with a "period" instead of a "exclamation point" ok. vs ok! The more we move to a text only world with less human face to face time, the more &nbsp;that understand text analytics is important not just for machines learning but for our intereactions with each other.&nbsp;</p>

<h2>How to Get Started</h2>

<h3>Requirements</h3>

<ul>
	<li>Mac OS or Linux</li>
	<li>Python 2.7</li>
	<li>DataStax Enterprise 6</li>
	<li>Jupyter (latest)</li>
	<li>Twitter Development API Access</li>
	<li>Python Packages: pandas, tweepy, cassandra-driver, pattern&nbsp;</li>
	<li>Notebook downloaded from: <a href="https://github.com/amandamoran/sentimentAnalysisDSE">https://github.com/amandamoran/sentimentAnalysisDSE</a></li>
</ul>

<p>This was all setup on a Mac OS laptop and all instructions should also work for a Linux machine. Other variations will work, but not by the instructions listed below.</p>

<h4>Twitter API Access</h4>

<p>Twitter Developer API access detailed instructions can be found here: <a href="https://developer.twitter.com/en/docs/ads/general/guides/getting-started.html">https://developer.twitter.com/en/docs/ads/general/guides/getting-started.html&nbsp;</a></p>

<p><img alt="First Steps" data-entity-type="file" data-entity-uuid="b8afbf59-7da7-4871-b94b-100e6a134734" src="https://www.datastax.com/sites/default/files/inline-images/firststep.png" /></p>

<p>You must create a Twitter app to be able to use the Twitter Dev API. It's a bit confusing the first time you create an app, but just walk through the instructions they provide and its actually straight forward.&nbsp;</p>

<p>Twitter has recently added an additional step that you must fill out an application for a Developer Account before creating the Twitter App. &nbsp;</p>

<p><img alt="Twitter Apps" data-entity-type="file" data-entity-uuid="296d84d0-9c67-45d8-b143-d804ee894fb8" src="https://www.datastax.com/sites/default/files/inline-images/Screen%20Shot%202018-09-21%20at%2012.33.03%20PM.png" /></p>

<p>Note: You can still run the demo with out Twitter API access. Another notebook "When Rotten Tomatoes isn’t Enough CSV.ipynb" with two CSV files is also included in github.&nbsp;</p>

<p>Once you get you authentication keys "Consumer Key, Consumer Secret, Access Token, and Access Token Secret" you will need to set these as environment variables.&nbsp;</p>

<p>In <code>~/.bashrc </code> (or on Mac <code>~/.bash_profile</code>):</p>

<p><img alt="Bash" data-entity-type="file" data-entity-uuid="de01bf1e-e4e7-4932-b01b-08fc5653e1a7" src="https://www.datastax.com/sites/default/files/inline-images/bash.png" /></p>

<h4>Install DataStax Enterprise</h4>

<p>Installing DataStax Enterprise on one node (even multiple nodes!) is actually very simple. &nbsp;This link has very complete installation instructions <a href="https://docs.datastax.com/en/install/doc/install60/installTOC.html">https://docs.datastax.com/en/install/doc/install60/installTOC.html&nbsp;</a></p>

<p>If installing on a Mac machine (or Linux) the instructions are detailed here: <a href="https://docs.datastax.com/en/install/doc/install60/installTARdse.html">https://docs.datastax.com/en/install/doc/install60/installTARdse.html&nbsp;</a></p>

<p>As long as you have a DataStax Academy account you can download the tar file from here: <a href="https://academy.datastax.com/downloads#dse">https://academy.datastax.com/downloads#dse</a>&nbsp;</p>

<p><img alt="DataStax Enterprise" data-entity-type="file" data-entity-uuid="e600e277-6af4-4b69-8f57-6a0d413b2d89" src="https://www.datastax.com/sites/default/files/inline-images/datastaxenterprise.png" /></p>

<p>Note: I used all the defaults and did not setup different locations for my data, logs, and Spark directories. This took around 20 minutes.&nbsp;</p>

<p>After the installation has completed note down the full path names to these two <code>pyspark.zip</code> files. These will need to be added to the notebook to be run later.</p>

<p><code>&lt;&gt;/dse-6.x.x/resources/spark/python/lib/pyspark.zip<br />
&lt;&gt;/dse-6.x.x/resources/spark/python/lib/py4j-0.10.4-src.zip</code></p>

<h3>Install Jupyter Notebooks</h3>

<p>Jupyter can be very easily installed via pip: <a href="http://jupyter.org/install%C2%A0">http://jupyter.org/install</a></p>

<p>Note: I installed jupyter locally, but it will work with any python based virtual environment like anaconda.</p>

<p><code>python -m pip install jupyter</code></p>

<h3>Install Python Packages</h3>

<p><code>​python -m pip install cassandra-driver<br />
python -m pip install​ tweepy<br />
python -m pip install pattern<br />
python -m pip install pandas</code></p>

<h3>Start DataStax Enterprise with Analytics</h3>

<p>To start a DSE Analytics Cluster, no added configuration needs to be done. Simply start with a <code>-k</code> to start DSE in analytics mode.</p>

<p><code>dse cassandra -k</code></p>

<h3>Start Jupyter</h3>

<p>The most unique element to the setup that is different from other Jupyter notebook installs is how Jupyter is started. To get all the needed enviroment variables needed to connect Jupyter to our DSE cluster, we must start the Jupyter notebooks as a DSE process. From here Jupyter will launch a local browser at <a href="http://localhost:8888/">http://localhost:8888</a></p>

<p><code>dse exec jupyter notebook</code></p>

<p>A token will be needed to sign in, review the standard out/log files to find this token for the first time login.</p>

<p><code>[I 13:03:07.074 NotebookApp] The Jupyter Notebook is running at:<br />
[I 13:03:07.074 NotebookApp] http://localhost:8888/?token=66d194ea4f0302a05d7adda7b74f1f1f5e89dd9be829785a<br />
[I 13:03:07.074 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).<br />
[C 13:03:07.075 NotebookApp]&nbsp;<br />
&nbsp; &nbsp;&nbsp;<br />
&nbsp; &nbsp; Copy/paste this URL into your browser when you connect for the first time,<br />
&nbsp; &nbsp; to login with a token:<br />
&nbsp; &nbsp; &nbsp; &nbsp; http://localhost:8888/?token=66d194ea4f0302a05d7adda7b74f1f1f5e89dd9be829785a&amp;token=66d194ea4f0302a05d7adda7b74f1f1f5e89dd9be829785a</code></p>

<h3>Navigate to Notebook</h3>

<p>Navigate to the notebook via the tree directory that has been launched.&nbsp;</p>

<p><img alt="Navigate to the Notebook" data-entity-type="file" data-entity-uuid="bba69955-54c4-444a-a509-d3ff500816ef" src="https://www.datastax.com/sites/default/files/inline-images/Screen%20Shot%202018-09-10%20at%203.45.50%20PM.png" /></p>

<h3>Running the Notebook</h3>

<p>Once the notebook is open, one last configuration must be done. In the 4th cell of the notebook, two enviroment variables must be set to the paths to the pyspark.zip files found above.&nbsp;</p>

<p><code>&lt;&gt;/dse-6.x.x/resources/spark/python/lib/pyspark.zip<br />
&lt;&gt;/dse-6.x.x/resources/spark/python/lib/py4j-0.10.4-src.zip</code></p>

<p>After that is complete it is time to run the notebook!!</p>

<p><img alt="Environment Variables" data-entity-type="file" data-entity-uuid="3ed65878-fc55-4493-a553-2ead64d811cb" src="https://www.datastax.com/sites/default/files/inline-images/Screen%20Shot%202018-09-10%20at%203.50.40%20PM.png" /></p>

<p>Click "Run" for each cell. Take time to understand what each cell is doing.</p>

<h3>Let's Explore</h3>

<p>Now its time to do the real work of a Data Scientist: Explore!</p>

<p>Change the movie title and search for new movies!&nbsp;</p>

<p>Was Rotten Tomatoes enough? Did you find out new insights that might have been hidden to you before? Data analytics does not have to be complicated!&nbsp;</p>

<h2>Stay Tuned for Part 2 and Part 3</h2>

<p>Part 2 will focus on how to get this all set up via Docker.</p>

<p>Part 3 will focus on a step-by-step describtion of each cell of the notebook.&nbsp;</p>

<h2>More Information</h2>

<div class="zdb-database-table">
<table>
	<thead>
		<tr>
			<th>Technology</th>
			<th>Link</th>
		</tr>
	</thead>
	<tbody>
		<tr>
			<td>Twitter Developer API</td>
			<td><a href="https://developer.twitter.com/en/docs/ads/general/guides/getting-started.html">https://developer.twitter.com/en/docs/ads/general/guides/getting-started.html</a></td>
		</tr>
		<tr>
			<td>Apache Cassandra&nbsp;</td>
			<td><a href="http://cassandra.apache.org/">http://cassandra.apache.org/</a></td>
		</tr>
		<tr>
			<td>DataStax Enterprise Analytics</td>
			<td><a href="https://www.datastax.com/products/datastax-enterprise-analytics">https://www.datastax.com/products/datastax-enterprise-analytics</a></td>
		</tr>
		<tr>
			<td>DataStax Python Driver</td>
			<td><a href="https://datastax.github.io/python-driver/">https://datastax.github.io/python-driver/</a></td>
		</tr>
		<tr>
			<td>Apache Spark</td>
			<td><a href="https://spark.apache.org/">https://spark.apache.org/</a></td>
		</tr>
		<tr>
			<td>Apache Spark MLlib</td>
			<td><a href="https://spark.apache.org/docs/latest/ml-guide.html">https://spark.apache.org/docs/latest/ml-guide.html</a></td>
		</tr>
		<tr>
			<td>Python Pandas</td>
			<td><a href="https://pandas.pydata.org/">https://pandas.pydata.org/</a></td>
		</tr>
		<tr>
			<td>Pattern (Python)</td>
			<td><a href="https://www.clips.uantwerpen.be/pages/pattern-en#sentiment">https://www.clips.uantwerpen.be/pages/pattern-en#sentiment</a></td>
		</tr>
		<tr>
			<td>Jupyter Notebooks</td>
			<td><a href="http://jupyter.org/">http://jupyter.org/</a></td>
		</tr>
		<tr>
			<td>DataStax Academy&nbsp;</td>
			<td><a href="https://academy.datastax.com/">https://academy.datastax.com/</a></td>
		</tr>
		<tr>
			<td>DataCon LA Presentation&nbsp;</td>
			<td><a href="http://www.youtube.com/watch?v=I8-bxR1Wew8%C2%A0">https://www.youtube.com/watch?v=I8-bxR1Wew8&nbsp;</a></td>
		</tr>
	</tbody>
</table>
</div>


When Rotten Tomatoes Isn't Enough: Twitter Sentiment Analysis with DSE Part 1

Amanda Moran

Discover more

Share

More Technology

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Amanda Moran

Discover more

Share

More Technology

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale