Tyler Hobbs

<p><a href="http://github.com/cloudera/flume">Flume</a>&nbsp;is an open source project described as a “distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data”. Because that large amount of log data eventually needs to be stored in a distributed, reliable, and available way, there was&nbsp;<a href="https://issues.cloudera.org/browse/FLUME-20">interest</a>&nbsp;in allowing Cassandra to be used as a destination for data, or “sink,” in Flume terminology.</p>

<p>A&nbsp;<a href="http://github.com/thobbs/flume-cassandra-plugin">new Cassandra plugin for Flume</a>&nbsp;is designed to require very little configuration and produce good results out of the box. The plugin offers two sinks: a simple sink that indexes entries by date, and a sink designed to take syslog events and store for use with the&nbsp;<a href="http://github.com/thobbs/logsandra">Logsandra</a>&nbsp;search engine, a tool for searching and analyzing logs stored in Cassandra.</p>

<p>&nbsp;</p>

<h3>Simple Cassandra Sink</h3>

<p>This sink makes use of two column families: one for storing the data, and one for storing and index. Upon receiving a Flume event, the Cassandra sink does the following:</p>

<ol>
	<li>Create a column where the name is a timestamp-based UUID and the value is empty. Insert this column into the index column family with row key&nbsp;<em>YYYYMMDDHH</em>&nbsp;(the current hour).</li>
	<li>Create a second column where the name is “data” and the value is the body of the Flume event. Insert this column into the data column family using the UUID from step #1 as the row key.</li>
</ol>

<p>The bulk of the workload -- storing the event bodies -- is evenly distributed around the Cassandra cluster, but the index column family still allows you to easily query for all events from a slice of time.</p>

<h3>Logsandra Syslog Sink</h3>

<p><a href="http://github.com/thobbs/logsandra">Logsandra</a>&nbsp;is a project that allows you to search logs by keyword and view graphs of their occurrence over time.</p>

<p>Assuming your Cassandra cluster is setup to work with Logsandra, the sink should be ready to use out of the box; all it requires is a list of Cassandra servers.</p>

<p>Syslog events are stored in Cassandra using a timestamp-based UUID as the row key and each event is indexed by its source, syslog facility, and syslog severity. A Logsandra query on any of these fields will show a list of all log events that match.</p>

<h3>License and Location</h3>

<p>The plugin is open source under the&nbsp;<a href="http://www.opensource.org/licenses/mit-license.html">MIT license</a>&nbsp;and is available here:</p>

<p>http://github.com/thobbs/flume-cassandra-plugin</p>


Cassandra Flume Sink and Logsandra Integration

Tyler Hobbs

Share

Share

Simple Cassandra Sink

Logsandra Syslog Sink

License and Location

More Company

DataStax Acquires Langflow to Accelerate Generative AI Development

The Top 5 DataStax Stories from 2023

2023 Recap: Data = AI

DataStax Astra DB Nabs Three Prestigious 2023 TrustRadius “Best of” Awards, Dominates the Vector Databases Category

One-stop Data API for Production GenAI