Headless Solrmeter with DataStax Enterprise
Not too long ago, I was tasked with getting an automated regression test in place for DataStax Enterprise’s bundled version of Apache Solr. After taking a bit of time to explore the various options available, it was decided that SolrMeter would be an excellent fit for the project, except for the fact that it did not have a headless mode of operation and so controlling it from the command line would not be an easy task. After spending a few hours digging into SolrMeter’s code I came to the conclusion that, owing to the excellent architecture of SolrMeter itself, it would not be too difficult of a job to create a patch that would allow it to run in headless mode and be drivable from the command line, which is what we needed for our project. So that is exactly what I did, and in this post I will show you how to get a headless SolrMeter up and running with very little effort. I hope that you will find this useful and possibly even be able to use this in your own automation projects.
For the examples that follow, I used a single node DataStax Enterprise cluster on EC2. The single and only node of this cluster has been started up as a solr/search node, and I have taken some extra steps to load data into Solr that we will access later with SolrMeter. The steps I took were as follows.
automaton@ip-10-172-89-53:~$ cd /usr/share/dse-demos/wikipedia/ automaton@ip-10-172-89-53:/usr/share/dse-demos/wikipedia$ ls 1-add-schema.sh 2-index.sh build.xml download-wiki.sh README.txt schema.xml set-solr-options.sh solrconfig.xml src wikipedia_import.jar wikipedia-sample.bz2 automaton@ip-10-172-89-53:/usr/share/dse-demos/wikipedia$ ./1-add-schema.sh SUCCESS Posted solrconfig.xml to http://ip-10-172-89-53.us-west-1.compute.internal:8983/solr/resource/wiki.solr/solrconfig.xml SUCCESS Posted schema.xml to http://ip-10-172-89-53.us-west-1.compute.internal:8983/solr/resource/wiki.solr/schema.xml 02880 Created index. automaton@ip-10-172-89-53:/usr/share/dse-demos/wikipedia$ ./2-index.sh --wikifile wikipedia-sample.bz2 args: [--host, ip-10-172-89-53.us-west-1.compute.internal, --scheme, http, --wikifile, wikipedia-sample.bz2] Start indexing wikipedia... ------------> config properties: docs.file = wikipedia-sample.bz2 keep.image.only.docs = false ------------------------------- 13/10/21 21:30:25 INFO impl.HttpClientUtil: Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false Indexed 1000 Indexed 2000 Indexed 3000 Finished
Note: The above steps can also found in the “Running The Solr Demo” section of DataStax’s “Getting Started with Cassandra and DataStax Enterpise” documentation. Here is a link to that section of the docs for additional context: Running the Solr Demo
After having followed the steps above, we can verify that everything went well by having a look at the admin page for the wiki.solr core. Mine was available at the following URL. Yours will of course be different.
From the screenshot below we can see that the wiki.solr core is now available and ready for us to use.
So that’s it for the initial setup, let’s now turn our focus towards getting the headless version of SolrMeter up and running.
I’ve submitted the patch that I created to the SolrMeter project for discussion/review. You can obtain it directly from the issue tracker which is found here: https://code.google.com/p/solrmeter/issues/detail?id=94. Make sure to get the most recent version of the patch: solrmeter-R318-headless-v4.patch. It should apply cleanly to revision 318 of SolrMeter, like so:
[~/solrmeter-headless/solrmeter-read-only] mallen-> ls solrmeter-R318-headless-v4.patch sources [~/solrmeter-headless/solrmeter-read-only] mallen-> patch -p1 < solrmeter-R318-headless-v4.patch patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/HeadlessModule.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/SolrMeterMain.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/SolrMeterRunModeModule.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/StatisticsModule.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/controller/StatisticDescriptor.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/model/SolrMeterConfiguration.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/model/statistic/HistogramQueryStatistic.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/model/statistic/QueryTimeHistoryStatistic.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/model/statistic/SimpleQueryStatistic.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/runMode/AbstractSolrMeterRunMode.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/runMode/SolrMeterRunMode.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/runMode/SolrMeterRunModeGUI.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/runMode/SolrMeterRunModeHeadless.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessCommitConsolePanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessConsoleFrame.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessConsolePanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessOptimizeConsolePanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessQueryConsolePanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessStatisticPanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessStatisticsContainer.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessUpdateConsolePanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/HeadlessUtils.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/QueryConsolePanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/statistic/HeadlessCacheHistoryPanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/statistic/HeadlessErrorLogPanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/statistic/HeadlessFullQueryStatisticPanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/statistic/HeadlessHistogramChartPanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/statistic/HeadlessOperationTimeLineChartPanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/statistic/HeadlessPieChartPanel.java patching file sources/solrmeter/src/main/java/com/plugtree/solrmeter/view/statistic/HeadlessQueryTimeHistoryPanel.java patching file sources/solrmeter/src/main/resources/StatisticDescriptorMapping.xml patching file sources/solrmeter/src/main/resources/solrmeter.properties patching file sources/solrmeter/src/main/resources/statistics-config.xml patching file sources/solrmeter/src/test/java/com/plugtree/solrmeter/controller/StatisticsRepositoryTestCase.java [~/solrmeter-headless/solrmeter-read-only] mallen->
After successfully applying the patch, change into the “sources” directory and run “mvn package” like so:
[~/solrmeter-headless/solrmeter-read-only/sources] mallen-> mvn package ... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Solrmeter parent POM .............................. SUCCESS [0.039s] [INFO] StressTestScope ................................... SUCCESS [28.294s] [INFO] SolrMeter ......................................... SUCCESS [54.670s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:23.204s [INFO] Finished at: Mon Oct 21 16:08:32 MDT 2013 [INFO] Final Memory: 28M/175M [INFO] ------------------------------------------------------------------------ [~/solrmeter-headless/solrmeter-read-only/sources]
That done, we can now run SolrMeter in GUI mode just as we could with the non-patched version:
[~/solrmeter-headless/solrmeter-read-only/sources] mallen-> java -jar ./solrmeter/target/solrmeter-0.3.1-SNAPSHOT-jar-with-dependencies.jar
With the patch applied and verification that SolrMeter starts up successfully in GUI mode, it’s time to get things setup to run in headless mode, which just takes a few more steps.
Setting up for Headless Operation
The first thing that we need to do to run in headless mode is get a valid SolrMeter configuration file. We can get that easily by running SolrMeter in GUI mode, editing the settings to point to the correct query and update URLs and to the correct query and update files, and then exporting the settings to a configuration file on the file system. We will use this config file later to start up in headless mode. Here are some screenshots of that process:
You’ll notice in the screenshots above that we are using a file named queries.txt for the queries file and one named updates.txt for the updates file. Below are the contents of those two files. Note that headless mode does not require this setup. Any of the normal operating modes of SolrMeter will work in headless mode too. Fields, filter, and facet files can all be used in headless mode as well.
Contents of queries.txt and updates.txt
[~/solrmeter-headless] mallen-> ls my-solrmeter-config.smc.xml queries.txt solrmeter-read-only updates.txt [~/solrmeter-headless] mallen-> cat queries.txt Weather "Space station" "Train station" Underground "Fort Knox" [~/solrmeter-headless] mallen-> cat updates.txt id:update-number-1;name:"update number 1";title:"title for update number 1";date:"Oct 21, 2013" id:update-number-2;name:"update number 2";title:"title for update number 2";date:"Oct 22, 2023" id:update-number-3;name:"update number 3";title:"title for update number 3";date:"Oct 23, 2033" id:update-number-4;name:"update number 4";title:"title for update number 4";date:"Oct 24, 2043" id:update-number-5;name:"update number 5";title:"title for update number 5";date:"Oct 25, 2053" [~/solrmeter-headless] mallen->
As you can see, we are using some very simple queries and updates in this example for demonstration purposes only. In practice, we use a more involved setup to achieve more useful results. OK, we now have everything we need in place to run SolrMeter in headless mode. Next, we’ll see exactly how to do that.
Running SolrMeter in Headless Mode
Running SolrMeter in headless mode from the command line is easily done by specifying and setting values for two properties: solrmeter.runMode and solrmeter.configurationFile. We’ll set the latter to point to the configuration file that we created earlier. The former, we’ll set to “headless” which will cause SolrMeter to start up in headless mode. So let’s go ahead and start it up now and see what happens!
java -Dsolrmeter.runMode=headless \ -Dsolrmeter.configurationFile=./my-solrmeter-config.smc.xml \ -jar target./solrmeter/target/solrmeter-0.3.1-SNAPSHOT-jar-with-dependencies.jar
The startup process appears pretty normal. Usual text gets output to the console as follows:
mallen-> java -Dsolrmeter.runMode=headless -Dsolrmeter.configurationFile=./my-solrmeter-config.smc.xml -jar solrmeter-read-only/sources/solrmeter/target/solrmeter-0.3.1-SNAPSHOT-jar-with-dependencies.jar [main]| INFO|[boot]|20:45:45,881 | Adding plugins from ./plugins [main]| WARN|[boot]|20:45:45,882 | No plugins directory found. No pluggin added [main]| INFO|[com.plugtree.solrmeter.SolrMeterMain]|20:45:45,922 | Using module: com.plugtree.solrmeter.StatisticsModule [main]| INFO|[org.exolab.castor.mapping.Mapping]|20:45:45,947 | Loading mapping descriptors from jar:file:/Users/mallen/solrmeter-headless/solrmeter-read-only/sources/solrmeter/target/solrmeter-0.3.1-SNAPSHOT-jar-with-dependencies.jar!/StatisticDescriptorMapping.xml [main]| INFO|[com.plugtree.solrmeter.SolrMeterMain]|20:45:46,213 | Using module: com.plugtree.solrmeter.ModelModule [main]| INFO|[com.plugtree.solrmeter.SolrMeterMain]|20:45:46,214 | Using module: com.plugtree.solrmeter.HeadlessModule [main]| INFO|[com.plugtree.solrmeter.SolrMeterMain]|20:45:46,215 | Using module: com.plugtree.solrmeter.SolrMeterRunModeModule [main]| INFO|[com.plugtree.solrmeter.model.SolrServerRegistry]|20:45:46,649 | Connecting to Solr: http://ec2-54-215-120-200.us-west-1.compute.amazonaws.com:8983/solr/wiki.solr [main]| INFO|[boot]|20:45:46,830 | Adding Statistic histogram [main]| INFO|[com.plugtree.solrmeter.view.I18n]|20:45:46,831 | Using Locale en_US [main]| INFO|[boot]|20:45:46,833 | Adding Statistic timeRange [main]| INFO|[boot]|20:45:46,835 | Adding Statistic queryHistory [main]| INFO|[boot]|20:45:46,835 | Adding Statistic operationHistory [main]| INFO|[boot]|20:45:46,836 | Adding Statistic fullQueryStatistic [main]| INFO|[boot]|20:45:46,836 | Adding Statistic errorLog [main]| INFO|[boot]|20:45:46,836 | Adding Statistic CacheStatistic [main]| INFO|[boot]|20:45:46,836 | Adding Statistic QueryLogStatistic [main]| INFO|[boot]|20:45:46,836 | Adding Statistic CommitHistoryStatistic [main]| INFO|[boot]|20:45:46,836 | Adding Statistic SimpleOptimizeStatistic [main]| INFO|[boot]|20:45:46,837 | Adding Statistic SimpleQueryStatistic [main]| INFO|[boot]|20:45:46,837 | Adding Statistic OperationRateStatistic
Take note, however, that SolrMeter is now using a couple of new modules that it didn’t used to: SolrMeterRunModeModule and HeadlessModule. That’s your indication that SolrMeter has found your configuration file and is running correctly in headless mode. Well, that and you will surely notice that the GUI doesn’t appear on the screen!
By default SolrMeter will execute 100 queries and then exit. It should take about 5 minutes to complete with the default of 20 queries per minute.(Both of these settings are configurable so feel free to run 5,000+ queries at 1,000 per minute, for examaple) By default no updates will be performed and no commits or optimizes either.
OK, 5 minutes hence and the process has finished and returned us to the console. We have successfully run SolrMeter in headless mode. But wait! That doesn’t do us much good if we can’t see the results of the operations that have been performed like we could in GUI mode, right? Where are the results?
The headless output directory
Unless the headless.outputDirectory property has been changed in the configuration file, running SolrMeter in headless mode will result in a directory being created named solrmeter-headless in the current working directory. This is where you will find all the statistics that you are used to seeing in GUI mode. Let’s have a look in there now.
mallen-> find solrmeter-headless solrmeter-headless solrmeter-headless/directives solrmeter-headless/query_console solrmeter-headless/statistics solrmeter-headless/statistics/error_log solrmeter-headless/statistics/histogram solrmeter-headless/statistics/operation_time_line solrmeter-headless/statistics/pie_chart solrmeter-headless/statistics/query_statistics solrmeter-headless/statistics/query_time_history [~/solrmeter-headless]
If you are familiar with SolrMeter, you will notice that the directory structure mostly mirrors that of the GUI component structure/layout. Most every component from the GUI version of SolrMeter has a counterpart in headless mode that maps conceptually as follows:
Main Window => solrmeter-headless
Query Console => solrmeter-headless/query_console
Query Statistics => solrmeter-headless/statistics/query_statistics
The information that you are used to seeing in GUI mode is available to you on the file system now in headless mode. The information in this directory refreshes continuously as operation results become available. You may have noticed that there is no update_console file. This is because no updates were performed. Let’s update our config file so that updates will be performed along with queries the next time we run SolrMeter in headless mode:
mallen-> cat my-solrmeter-config.smc.xml | grep headless | grep -v "guice" <entry key="headless.performQueryOperations">true</entry> <entry key="headless.numUpdates">100</entry> <entry key="headless.outputDirectory">./solrmeter-headless</entry> <entry key="headless.performUpdateOperations">false</entry> <entry key="headless.numQueries">100</entry> [~/solrmeter-headless] mallen->
Above are the settings relevant to headless mode that you can adjust.
Edit my-solrmeter-config.smc.xml and change performUpdateOperations from “false” to “true”. Now, we run SolrMeter again in headless mode just like we did before. Another 5 minutes or so later, we inspect the solrmeter-headless directory again:
mallen-> find solrmeter-headless solrmeter-headless solrmeter-headless/commit_console solrmeter-headless/directives solrmeter-headless/query_console solrmeter-headless/statistics solrmeter-headless/statistics/error_log solrmeter-headless/statistics/histogram solrmeter-headless/statistics/operation_time_line solrmeter-headless/statistics/pie_chart solrmeter-headless/statistics/query_statistics solrmeter-headless/statistics/query_time_history solrmeter-headless/update_console [~/solrmeter-headless]
Largely the same as before but this time you’ll notice that there are two new files present: update_console and commit_console. These two new files contain the results of those operations (updates and commits).
Everything seems to be operating as it should be. Let’s take a look now at the contents of some of the files in the output directory to see what information is available to us.
mallen-> cat query_console queries per minute: 20 total queries: 101 total query time: 59 total client time: 331542 total errors: 0 average query time: 0 average client time: 3282 intended queries per minute: 20 actual query rate: 36.0 [~/solrmeter-headless/solrmeter-headless] mallen-> cat update_console added documents: 100 errors on update: 0 intended updates per minute: 20 update rate: 12.0 [~/solrmeter-headless/solrmeter-headless] mallen-> cat commit_console not committed documents: 2 last commit: 10/21/13 9:23 PM total commits: 29 errors on commit: 0
As you can see from above, the headless mode of operation attempts to make the same information that is available to you in GUI mode available to you in headless mode too. Excellent!
Well, that about covers it, but there is just one last operation that we haven’t talked about yet. How do you perform an optimize in headless mode, like you do in GUI mode? In GUI mode, when you want to perform an optimize operation you click the “optimize now” button. In headless mode there are no buttons to push but we can still achieve our goal of ‘on-demand’ optimize operations as follows. When in headless mode, SolrMeter will listen for changes to the “directives” directory. This is our way of communicating with SolrMeter. When you want to perform an optimize operation in headless mode create a file in that directory named “optimize”. Example:
mallen-> touch directives/optimize
SolrMeter will then schedule on optimize operation to be performed and delete that file. You can repeat this process as often as you’d like to. Take note that after an optimize operation has been performed you will then see a new file created inside the solrmeter-headless directory named (you guessed it!) optimize_console, which is the counterpart of the “Optimze Console” panel from the GUI version.
I hope that you have found the information in this post useful and/or interesting. We have been using the headless mode of SolrMeter for a few months now with good results in our test harness here at DataStax. On a personal note, I enjoyed working on the patch for SolrMeter and I want to thank the members of that project for providing such a useful tool and a well designed system, which ultimately made my job easier!