DataStax Support Forums » Topic: Suggestions to write DSE Solr unit test cases .

DataStax Support Forums » Topic: Suggestions to write DSE Solr unit test cases . http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases Software, Support, and Training for Apache Cassandra en-US Fri, 24 May 2013 02:40:30 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php shruti on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7583 Tue, 20 Nov 2012 09:15:32 +0000 shruti 7583@http://www.datastax.com/support-forums/ Hi Sam , I have tried and integrated the Cassandra-unit using DSE jars . I am able to start DSE cassandra and load data into it using cassandra-unit , however when posting the solr xml files I am getting exception - SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/C:/Documents%20and%20Settings/sm40/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/C:/dse-2.2/resources/dse/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" rel="nofollow">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation. Catalina Base----->src/test/resources/tomcat/ 2012-11-20 14:40:37,577 [main] DEBUG cassandraunit.utils.EmbeddedCassandraServerHelper - Starting cassandra... 2012-11-20 14:40:37,577 [main] DEBUG cassandraunit.utils.EmbeddedCassandraServerHelper - Initialization needed 2012-11-20 14:40:37,655 [Thread-1] DEBUG hadoop.conf.Configuration - java.io.IOException: config() at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:228) at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:215) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:321) at com.datastax.bdp.hadoop.mapred.CassandraJobConf.<init>(CassandraJobConf.java:32) at com.datastax.bdp.server.DseServer.<clinit>(DseServer.java:122) at com.datastax.bdp.hadoop.mapred.SchemaTool.init(SchemaTool.java:408) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:113) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:389) at org.cassandraunit.utils.EmbeddedCassandraServerHelper$1.run(EmbeddedCassandraServerHelper.java:106) at java.lang.Thread.run(Unknown Source) 2012-11-20 14:40:38,015 [Thread-1] INFO bdp.snitch.DseDelegateSnitch - Detected enabled Solr plugin, setting my role to Search 2012-11-20 14:40:38,030 [Thread-1] INFO bdp.config.DseConfig - Loading settings from file:/D:/cassandra-unit-master/target/test-classes/dse.yaml 2012-11-20 14:40:38,030 [Thread-1] DEBUG bdp.config.DseConfig - Delegated snitch is com.datastax.bdp.snitch.DseSimpleSnitch 2012-11-20 14:40:38,030 [Thread-1] INFO bdp.snitch.DseSimpleSnitch - Detected search service is enabled, setting my DC to Solr 2012-11-20 14:40:38,030 [Thread-1] INFO bdp.config.DseConfig - Load of settings is done. 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - DSE version: 2.2 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Hadoop version: 1.0.2-dse-SNAPSHOT 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Hive version: 0.9.0-SNAPSHOT 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Pig version: 0.9.2 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Solr version: 4.0-SNAPSHOT 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Sqoop version: 1.4.2-dse-SNAPSHOT 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Mahout version: 0.6 2012-11-20 14:40:38,374 [Thread-1] INFO bdp.server.DseDaemon - Appender version: 1.0-SNAPSHOT 2012-11-20 14:40:39,311 [pool-1-thread-1] INFO bdp.plugin.SolrContainerPlugin - Starting Embedded Tomcat Nov 20, 2012 2:40:39 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jre6\bin;.;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:/Program Files/Java/jre6/bin/client;C:/Program Files/Java/jre6/bin;C:\Program Files\PC Connectivity Solution\;C:\MINGW\bin;C:\Perl\site\bin;C:\Perl\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Perforce;C:\Program Files\MATLAB\R2010a\bin;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Program Files\Enterprise Vault\EVClient\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\Ant\bin;D:\Program Files\Perforce;C:\Program Files\Windows Imaging\;C:\ostaw32-1-4-4\ext\omniorb\bin\x86_win32;C:\gnuwin32\bin;C:\Program Files\Microsoft Visual Studio 8\VC\bin;C:\Program Files\Microsoft Visual Studio 8\Common7\IDE;C:\Program Files\Microsoft Visual Studio 8\VC\include;C:\Program Files\Python;C:\Program Files\Microsoft Visual Studio 8\VC\redist\x86\Microsoft.VC80.CRT;C:\cygwin\bin;C:\project\android-ndk-r6b;C:\Program Files\Java\jdk1.6.0_25\bin;C:\apache-maven-3.0.4\bin;C:\curl; 2012-11-20 14:40:39,702 [pool-1-thread-1] INFO org.mortbay.log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog Nov 20, 2012 2:40:39 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-8983 Nov 20, 2012 2:40:39 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 494 ms Nov 20, 2012 2:40:39 PM org.apache.catalina.core.StandardService start INFO: Starting service Solr Nov 20, 2012 2:40:39 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.32 Nov 20, 2012 2:40:39 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8983 Nov 20, 2012 2:40:39 PM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:8009 Nov 20, 2012 2:40:40 PM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/46 config=null 2012-11-20 14:40:40,014 [pool-1-thread-1] INFO bdp.plugin.SolrContainerPlugin - Embedded Tomcat started 2012-11-20 14:40:40,014 [pool-1-thread-1] INFO bdp.plugin.PluginManager - Plugin activated: com.datastax.bdp.plugin.SolrContainerPlugin Nov 20, 2012 2:40:40 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 157 ms 2012-11-20 14:40:40,061 [Thread-1] INFO bdp.server.DseDaemon - Binding thrift service to /0.0.0.0:9171 2012-11-20 14:40:40,061 [Thread-1] INFO bdp.server.DseDaemon - Using TFastFramedTransport with a max frame size of 15728640 bytes. 2012-11-20 14:40:40,077 [Thread-1] INFO bdp.server.DseDaemon - Using synchronous/threadpool thrift server on /0.0.0.0 : 9171 2012-11-20 14:40:40,077 [Thread-9] INFO bdp.server.DseDaemon - Listening for thrift clients... 2012-11-20 14:40:41,124 [main] INFO org.cassandraunit.DataLoader - creating keyspace : testKeyspace 2012-11-20 14:40:41,124 [main] INFO org.cassandraunit.DataLoader - loading data into keyspace : testKeyspace shouldWork Get DataSet -->testKeyspace solrConfig URL->http://localhost:8983/solr/resource/testKeyspace.testcf/solrconfig.xml 2012-11-20 14:41:06,294 [main] INFO test.lts.PostTool - Entering PostTool postFile 2012-11-20 14:41:06,294 [main] INFO test.lts.PostTool - PostTool: POSTing files to <a href="http://localhost:8983/solr/resource/testKeyspace.testcf/solrconfig.xml." rel="nofollow">http://localhost:8983/solr/resource/testKeyspace.testcf/solrconfig.xml.</a>. 2012-11-20 14:41:06,294 [main] INFO test.lts.PostTool - PostTool: POSTing file solrconfig.xml java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(Unknown Source) at org.cassandraunit.test.lts.PostTool.pipe(PostTool.java:275) at org.cassandraunit.test.lts.PostTool.postData(PostTool.java:218) at org.cassandraunit.test.lts.PostTool.postFile(PostTool.java:138) at org.cassandraunit.test.lts.PostTool.postFiles(PostTool.java:80) at org.cassandraunit.test.lts.PostTool.postFile(PostTool.java:61) at org.cassandraunit.test.lts.LoadDeviceData.postSolrConfig(LoadDeviceData.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) 2012-11-20 14:41:15,466 [main] ERROR test.lts.PostTool - PostTool: FATAL: IOException while posting data: java.io.IOException: Error writing request body to server 2012-11-20 14:41:18,450 [main] ERROR test.lts.PostTool - PostTool: FATAL: IOException while reading response: java.io.IOException: Incomplete output stream Hope you can provide a pointer to solve this . Thanks ! sam on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7570 Mon, 19 Nov 2012 09:31:22 +0000 sam 7570@http://www.datastax.com/support-forums/ Are there any stacktraces printed to the console or the application log? shruti on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7566 Mon, 19 Nov 2012 06:06:45 +0000 shruti 7566@http://www.datastax.com/support-forums/ hi Sam , The exception occurs when addSolrConfig method gets invoked . sam on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7495 Wed, 14 Nov 2012 09:54:30 +0000 sam 7495@http://www.datastax.com/support-forums/ If there are errors submitting the Solr resources, you should see a stacktrace and the test should immediately fail (this should happen during setup). Is this when you're seeing the 400 responses, or is it during the actual test execution? Are there any stacktraces printed to the console? The example should be self contained, so there isn't anything else you should need to configure for Solr. shruti on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7427 Wed, 07 Nov 2012 06:42:08 +0000 shruti 7427@http://www.datastax.com/support-forums/ Hi Sam , Using your code , I was able to start EmbeddedDSE . Cassandra has started properly as I was able to load data into it , However seems like Solr has not started as posting my solrconfig and schema xml's is throwing 400 Response Code . Do I need to provide some additional information /configuration for Solr to start up ? Thanks! shruti on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7395 Mon, 05 Nov 2012 08:37:40 +0000 shruti 7395@http://www.datastax.com/support-forums/ Thanks again Sam, However I am facing problems related to avro while creating the Keyspace .I am using avro 1.6 jar & avro-tools 1.6. sam on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7386 Sat, 03 Nov 2012 13:45:27 +0000 sam 7386@http://www.datastax.com/support-forums/ In the gist, lines 151-162 are adding data into the columnfamily. You could pull that out into a fixture setup method if you have a particular dataset that you'll use across tests. Thanks, Sam shruti on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7385 Sat, 03 Nov 2012 12:20:56 +0000 shruti 7385@http://www.datastax.com/support-forums/ Hi sam , Thanks a lot for the pointers . Just wanted to know how do you add data into your keyspace and columnfamily . Whenever I try and index a CF , it throws unconfigured CF exception . Is there a way populate data in cassandra daemon which gets started . sam on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7124 Fri, 26 Oct 2012 10:04:26 +0000 sam 7124@http://www.datastax.com/support-forums/ Whilst it isn't something maintained or supported by DataStax, this is an approach that you can use to run an embedded DSE Search instance in junit tests. It certainly isn't as polished as cassandra-unit, but its something which has worked well for me <a href="https://gist.github.com/344d9e60be76b6c6df58" rel="nofollow">https://gist.github.com/344d9e60be76b6c6df58</a> Hope you find it useful rahja on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7092 Wed, 24 Oct 2012 07:17:18 +0000 rahja 7092@http://www.datastax.com/support-forums/ sdelmas, thanks for you reply. We have a few more questions related to unit testing with Solr. We intend to have a completely automated unit tests which can be easily run from our build systems or from developers machine, without requiring any additional/manual setups. For this we need a way to 1) Start DSE cassandra+solr, as embedded service in the same JVM as unit tests (junit). Apache cassandra supports running it as embedded service, like what is mentioned @ <a href="http://prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/" rel="nofollow">http://prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/</a> How do we start DSE cassandar+solr as embedded serrvice. For plain cassandra unit tests we are currently using cassandra-unit (<a href="https://github.com/jsevellec/cassandra-unit)" rel="nofollow">https://github.com/jsevellec/cassandra-unit)</a>, which depends on apache cassandra-all, and starts plain apache cassandra as embedded service. To start DSE cassandra+solr as embedded service, do we need to compile against any specific DSE jars? 2) How to populate data in embedded cassandra+solr and also instruct solr for indexing. Cassandra-Unit allows populating data very easily by defining them in XML/Json/Yaml data files? Is there any similar or other way with DSE cassandar+solr? Sven on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7088 Tue, 23 Oct 2012 20:43:36 +0000 Sven 7088@http://www.datastax.com/support-forums/ Solr is already in the same JVM as Cassandra when you use DSE. A good way to test Solr would be to use cql or http requests. You could use scripts to drive that or use an existing test framework like junit. shruti on "Suggestions to write DSE Solr unit test cases ." http://www.datastax.com/support-forums/topic/suggestions-to-write-dse-solr-unit-test-cases#post-7068 Tue, 23 Oct 2012 05:39:13 +0000 shruti 7068@http://www.datastax.com/support-forums/ Hi , Can you please suggest a way to write unit test cases using DSE Solr . Currently my application is storing data in DSE cassandra and querying it through Solr . Is there any way to start DSE in embedded/in memory mode so that it starts both Solr and Cassandra . Thanks!