not a support question
shruti
Member
Hi ,
Can you please suggest a way to write unit test cases using DSE Solr .
Currently my application is storing data in DSE cassandra and querying it through Solr .
Is there any way to start DSE in embedded/in memory mode so that it starts both Solr and Cassandra .
Thanks!
Solr is already in the same JVM as Cassandra when you use DSE. A good way to test Solr would be to use cql or http requests. You could use scripts to drive that or use an existing test framework like junit.
rahja
Member
sdelmas, thanks for you reply.
We have a few more questions related to unit testing with Solr. We intend to have a completely automated unit tests which can be easily run from our build systems or from developers machine, without requiring any additional/manual setups.
For this we need a way to
1) Start DSE cassandra+solr, as embedded service in the same JVM as unit tests (junit). Apache cassandra supports running it as embedded service, like what is mentioned @ http://prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/
How do we start DSE cassandar+solr as embedded serrvice.
For plain cassandra unit tests we are currently using cassandra-unit (https://github.com/jsevellec/cassandra-unit), which depends on apache cassandra-all, and starts plain apache cassandra as embedded service.
To start DSE cassandra+solr as embedded service, do we need to compile against any specific DSE jars?
2) How to populate data in embedded cassandra+solr and also instruct solr for indexing. Cassandra-Unit allows populating data very easily by defining them in XML/Json/Yaml data files? Is there any similar or other way with DSE cassandar+solr?
Whilst it isn't something maintained or supported by DataStax, this is an approach that you can use to run an embedded DSE Search instance in junit tests. It certainly isn't as polished as cassandra-unit, but its something which has worked well for me
https://gist.github.com/344d9e60be76b6c6df58
Hope you find it useful
shruti
Member
Hi Sam ,
Using your code , I was able to start EmbeddedDSE . Cassandra has started properly as I was able to load data into it , However seems like Solr has not started as posting my solrconfig and schema xml's is throwing 400 Response Code .
Do I need to provide some additional information /configuration for Solr to start up ?
Thanks!
If there are errors submitting the Solr resources, you should see a stacktrace and the test should immediately fail (this should happen during setup). Is this when you're seeing the 400 responses, or is it during the actual test execution?
Are there any stacktraces printed to the console?
The example should be self contained, so there isn't anything else you should need to configure for Solr.
shruti
Member
Hi Sam ,
I have tried and integrated the Cassandra-unit using DSE jars . I am able to start DSE cassandra and load data into it using cassandra-unit , however when posting the solr xml files I am getting exception -
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Documents%20and%20Settings/sm40/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/dse-2.2/resources/dse/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Catalina Base----->src/test/resources/tomcat/
2012-11-20 14:40:37,577 [main] DEBUG cassandraunit.utils.EmbeddedCassandraServerHelper - Starting cassandra...
2012-11-20 14:40:37,577 [main] DEBUG cassandraunit.utils.EmbeddedCassandraServerHelper - Initialization needed
2012-11-20 14:40:37,655 [Thread-1] DEBUG hadoop.conf.Configuration - java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:228)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:215)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:321)
at com.datastax.bdp.hadoop.mapred.CassandraJobConf.<init>(CassandraJobConf.java:32)
at com.datastax.bdp.server.DseServer.<clinit>(DseServer.java:122)
at com.datastax.bdp.hadoop.mapred.SchemaTool.init(SchemaTool.java:408)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:113)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:389)
at org.cassandraunit.utils.EmbeddedCassandraServerHelper$1.run(EmbeddedCassandraServerHelper.java:106)
at java.lang.Thread.run(Unknown Source)
2012-11-20 14:40:38,015 [Thread-1] INFO bdp.snitch.DseDelegateSnitch - Detected enabled Solr plugin, setting my role to Search
2012-11-20 14:40:38,030 [Thread-1] INFO bdp.config.DseConfig - Loading settings from file:/D:/cassandra-unit-master/target/test-classes/dse.yaml
2012-11-20 14:40:38,030 [Thread-1] DEBUG bdp.config.DseConfig - Delegated snitch is com.datastax.bdp.snitch.DseSimpleSnitch
2012-11-20 14:40:38,030 [Thread-1] INFO bdp.snitch.DseSimpleSnitch - Detected search service is enabled, setting my DC to Solr
2012-11-20 14:40:38,030 [Thread-1] INFO bdp.config.DseConfig - Load of settings is done.
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - DSE version: 2.2
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Hadoop version: 1.0.2-dse-SNAPSHOT
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Hive version: 0.9.0-SNAPSHOT
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Pig version: 0.9.2
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Solr version: 4.0-SNAPSHOT
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Sqoop version: 1.4.2-dse-SNAPSHOT
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Mahout version: 0.6
2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Appender version: 1.0-SNAPSHOT
2012-11-20 14:40:39,311 [pool-1-thread-1] INFO bdp.plugin.SolrContainerPlugin - Starting Embedded Tomcat
Nov 20, 2012 2:40:39 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jre6\bin;.;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:/Program Files/Java/jre6/bin/client;C:/Program Files/Java/jre6/bin;C:\Program Files\PC Connectivity Solution\;C:\MINGW\bin;C:\Perl\site\bin;C:\Perl\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Perforce;C:\Program Files\MATLAB\R2010a\bin;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Program Files\Enterprise Vault\EVClient\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\Ant\bin;D:\Program Files\Perforce;C:\Program Files\Windows Imaging\;C:\ostaw32-1-4-4\ext\omniorb\bin\x86_win32;C:\gnuwin32\bin;C:\Program Files\Microsoft Visual Studio 8\VC\bin;C:\Program Files\Microsoft Visual Studio 8\Common7\IDE;C:\Program Files\Microsoft Visual Studio 8\VC\include;C:\Program Files\Python;C:\Program Files\Microsoft Visual Studio 8\VC\redist\x86\Microsoft.VC80.CRT;C:\cygwin\bin;C:\project\android-ndk-r6b;C:\Program Files\Java\jdk1.6.0_25\bin;C:\apache-maven-3.0.4\bin;C:\curl;
2012-11-20 14:40:39,702 [pool-1-thread-1] INFO org.mortbay.log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Nov 20, 2012 2:40:39 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8983
Nov 20, 2012 2:40:39 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 494 ms
Nov 20, 2012 2:40:39 PM org.apache.catalina.core.StandardService start
INFO: Starting service Solr
Nov 20, 2012 2:40:39 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
Nov 20, 2012 2:40:39 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8983
Nov 20, 2012 2:40:39 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Nov 20, 2012 2:40:40 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/46 config=null
2012-11-20 14:40:40,014 [pool-1-thread-1] INFO bdp.plugin.SolrContainerPlugin - Embedded Tomcat started
2012-11-20 14:40:40,014 [pool-1-thread-1] INFO bdp.plugin.PluginManager - Plugin activated: com.datastax.bdp.plugin.SolrContainerPlugin
Nov 20, 2012 2:40:40 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 157 ms
2012-11-20 14:40:40,061 [Thread-1] INFO bdp.server.DseDaemon - Binding thrift service to /0.0.0.0:9171
2012-11-20 14:40:40,061 [Thread-1] INFO bdp.server.DseDaemon - Using TFastFramedTransport with a max frame size of 15728640 bytes.
2012-11-20 14:40:40,077 [Thread-1] INFO bdp.server.DseDaemon - Using synchronous/threadpool thrift server on /0.0.0.0 : 9171
2012-11-20 14:40:40,077 [Thread-9] INFO bdp.server.DseDaemon - Listening for thrift clients...
2012-11-20 14:40:41,124 [main] INFO org.cassandraunit.DataLoader - creating keyspace : testKeyspace
2012-11-20 14:40:41,124 [main] INFO org.cassandraunit.DataLoader - loading data into keyspace : testKeyspace
shouldWork Get DataSet -->testKeyspace
solrConfig URL->http://localhost:8983/solr/resource/testKeyspace.testcf/solrconfig.xml
2012-11-20 14:41:06,294 [main] INFO test.lts.PostTool - Entering PostTool postFile
2012-11-20 14:41:06,294 [main] INFO test.lts.PostTool - PostTool: POSTing files to http://localhost:8983/solr/resource/testKeyspace.testcf/solrconfig.xml..
2012-11-20 14:41:06,294 [main] INFO test.lts.PostTool - PostTool: POSTing file solrconfig.xml
java.io.IOException: Error writing request body to server
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(Unknown Source)
at org.cassandraunit.test.lts.PostTool.pipe(PostTool.java:275)
at org.cassandraunit.test.lts.PostTool.postData(PostTool.java:218)
at org.cassandraunit.test.lts.PostTool.postFile(PostTool.java:138)
at org.cassandraunit.test.lts.PostTool.postFiles(PostTool.java:80)
at org.cassandraunit.test.lts.PostTool.postFile(PostTool.java:61)
at org.cassandraunit.test.lts.LoadDeviceData.postSolrConfig(LoadDeviceData.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
2012-11-20 14:41:15,466 [main] ERROR test.lts.PostTool - PostTool: FATAL: IOException while posting data: java.io.IOException: Error writing request body to server
2012-11-20 14:41:18,450 [main] ERROR test.lts.PostTool - PostTool: FATAL: IOException while reading response: java.io.IOException: Incomplete output stream
Hope you can provide a pointer to solve this .
Thanks !
You must log in to post.
