The Challenge

The leadership team at Hornet wanted to support the community that had formed around the app, expanding into new services and support approaches that would help users connect with each other around shared interests, locations, and topics. This meant looking at how some of the app’s most popular features functioned.

Matthew Hirst, Head of Server Side Engineering at Hornet, led the team involved in this project. “We estimated that the volume of messages that we would be supporting would go up by around fifty percent, and alongside the continued growth in the number of people using the app, that would become a problem in the future. We knew we had to make some changes to how we managed our data over time.”

With a focus on community development and more conversations taking place, the Hornet team knew that they would have to look at scaling up the existing messaging clusters. Hornet had implemented Apache CassandraTM previously, but this was a version that required an update. “Cassandra was what we originally chose for our messaging cluster, and it functioned superbly for that. We had no downtime and it just worked. However, that meant that it had been in place for a long time, and we had not updated our implementation. Alongside this, we looked at our storage and we were continuing to scale up the volume of data that we had. That was becoming an issue too, particularly based on the growth plans we had. We knew we needed to do something to prevent issues in the future,” explained Hirst.

The Solution

Hirst worked with Nate Mitchell, Lead DevOps Engineer at Hornet, on how the team would approach the project. The first step was to find the right consulting support for the upgrade. Hirst and Mitchell turned to DataStax for help.

“When we discussed this with the Cassandra community, there was some consistent feedback on talking to the team at DataStax. The team there has supported some of the largest implementations worldwide, so they have a lot of experience in dealing with issues like ours. At the same time, they were approachable right from the start,” explained Mitchell.

DataStax helped the Hornet team with designing its upgrade approach as well as optimizing its approach to managing clusters. DataStax also provided consulting advice on how to track the results of the shift, so that the team could demonstrate that it had taken the right approach.

“We had taken some design decisions in the past that made sense previously, but they did not fit with what the company wanted to achieve in the future. As part of the update to Cassandra on our messaging cluster, we were able to improve our efficiency around how we managed to archive messages. Not only would this deliver a reduction in the overall volume of storage that we would have in place, but it would also allow us to show more messages between users as standard,” added Mitchell.

The Results

Following the consulting sessions, the Hornet team worked with DataStax to implement the upgrade and management changes successfully. This was done in a phased approach, with the newsfeed cluster updated first, followed by the messaging cluster. This enabled the Hornet team to progress things at the right pace for them.

“We are a small team, so having the right advice and support on hand during the migrations helped us to achieve what we wanted. We also felt that the team at DataStax was invested in our success,” commented Hirst.

The Hornet team compared its approach before and after the migration and was pleased with the results. “We were able to achieve a 50 percent reduction in the volume of data that we had to store overall – this is a huge saving for us as a company, and it also helps us to support the long-term goals we have as a business around more community interaction and development. We completed our migrations successfully with no downtime for users, we are now on a version of Cassandra that will support our growth successfully well into the future, and we have that long-term support from DataStax available to us whenever we may need it too,” added Hirst.

The Hornet technology team is already looking to the future and making more use of Kubernetes for its application stack. Mitchell and Hirst have already looked at the work that DataStax is doing around bringing Cassandra and Kubernetes together in the K8ssandra open source project: “We think that everything will be based on Kubernetes, so we are evaluating our approach here as well.The fact that DataStax is supporting this with its open source K8ssandra distribution is great for us, and we are looking forward to exploring how this might help us in future,” commented Mitchell.