DataStax Support Forums » Tag: lucene

DataStax Support Forums » Tag: lucene - Recent Posts http://www.datastax.com/support-forums/tags/lucene Software, Support, and Training for Apache Cassandra en-US Wed, 22 May 2013 16:59:42 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php jas on "Solr Index Files" http://www.datastax.com/support-forums/topic/solr-index-files#post-1886 Wed, 09 May 2012 21:26:08 +0000 jas 1886@http://www.datastax.com/support-forums/ Thanks, Rock, that does help. I like to know where I stand. :) BTW, I see you're a forum "Member" like me as opposed to a "Moderator". Do you work for DataStax or just so happen to know these details? :) Thanks, Jeff Anonymous on "Solr Index Files" http://www.datastax.com/support-forums/topic/solr-index-files#post-1878 Wed, 09 May 2012 15:39:58 +0000 Anonymous 1878@http://www.datastax.com/support-forums/ Jeff, Your analysis of how the DSE Solr product works is correct. The index files are stored locally. This produces the maximum query speed possible, and enables taking advantage of new Lucene posting codecs. Optimization of the Lucene / Solr index will help in the read-only case. If updates are frequent, then optimizing will not be beneficial. > Does every node in the ring index the same content and product its own index files? Yes Hope that helps! jas on "Solr Index Files" http://www.datastax.com/support-forums/topic/solr-index-files#post-1865 Mon, 07 May 2012 21:14:43 +0000 jas 1865@http://www.datastax.com/support-forums/ Hi: For whatever reason I thought the DSE 2 Solr indexes were maintained in Cassandra. But in this document: <a href="http://www.datastax.com/docs/datastax_enterprise2.0/search/dse_search_cluster#updating-individual-fields-in-a-solr-document" rel="nofollow">http://www.datastax.com/docs/datastax_enterprise2.0/search/dse_search_cluster#updating-individual-fields-in-a-solr-document</a> Under "Manage the Location of Solr Data", describes the existence of <code>solr.data</code>. I went there and I see what look like numerous Lucene index files. This indicates to me that the indexes reside within the node's local file system, and not somewhere in Cassandra. I know the equivalently named keyspace and CF in Cassandra has the field data, and the Solr configuration files. Does <code>com.datastax.bdp.cassandra.index.solr.SolrSecondaryIndex</code> place the Lucene indexes themselves in the file system rather than in Cassandra itself? This means the indexes themselves are not distributed and replicated? Does every node in the ring index the same content and product its own index files? I guess that's a form of replication, but it could well exceed the desired replication factor. If I add a node, does it then have to go and index the Solr related CFs? If the indexes are indeed in the file system, then optimizing the index is still useful, right? I have a lot of read-only data indexed, and optimizing pays off well in regular Solr. I figured Cassandra's compaction took care of that, but I guess if the indexes don't reside there, that's not the case. Thanks, Jeff tjake on "Lucene/Mahout Error: class overrides final method tokenStream." http://www.datastax.com/support-forums/topic/lucenemahout-error-class-overrides-final-method-tokenstream#post-1561 Tue, 10 Apr 2012 14:06:09 +0000 tjake 1561@http://www.datastax.com/support-forums/ This looks to be happening because DSE comes with Lucene/Solr 4.0 We can fix this in the next point release so the Solr jars aren't in the classpath when you run hadoop. In the meantime you can move those jars out of the way $DSE_HOME/resources/solr/lib -Jake Anonymous on "Lucene/Mahout Error: class overrides final method tokenStream." http://www.datastax.com/support-forums/topic/lucenemahout-error-class-overrides-final-method-tokenstream#post-1544 Fri, 06 Apr 2012 15:33:24 +0000 Anonymous 1544@http://www.datastax.com/support-forums/ Hey, I've been working on getting Mahout to run on top of DataStax Enterprise, but I'm running into issues with Lucene incompatibility. For example, using the Mahout wikipedia bayes example (<a href="https://cwiki.apache.org/MAHOUT/wikipedia-bayes-example.html)" rel="nofollow">https://cwiki.apache.org/MAHOUT/wikipedia-bayes-example.html)</a>, when I execute the following command (step #6 from the example), I end up with errors as shown. (the below command has some *magic* in it, as mahout will assume that a <code>hadoop</code> executable will exist, so I created a <code>hadoop</code> shell script serving as an alias that basically does <code>dse hadoop $@</code>; that's what <code>HADOOP_HOME</code> is pointing to <code> sudo env JAVA_HOME=$JAVA_HOME HADOOP_HOME=/home/ubuntu/hadoop HADOOP_CONF_DIR=/etc/dse/hadoop ./mahout wikipediaDataSetCreator -i wikipedia/chunks -o wikipediainput -c /home/ubuntu/mahout/examples/src/test/resources/country.txt </code> <code> MAHOUT-JOB: /home/ubuntu/mahout/examples/target/mahout-examples-0.7-SNAPSHOT-job.jar 12/04/06 14:38:03 WARN driver.MahoutDriver: No wikipediaDataSetCreator.props found on classpath, will use command-line arguments only 12/04/06 14:38:04 INFO bayes.WikipediaDatasetCreatorDriver: Input: wikipedia/chunks Out: wikipediainput Categories: /home/ubuntu/mahout/examples/src/test/resources/country.txt 12/04/06 14:38:04 INFO common.HadoopUtil: Deleting wikipediainput 12/04/06 14:38:05 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 12/04/06 14:38:06 INFO input.FileInputFormat: Total input paths to process : 555 12/04/06 14:38:07 INFO mapred.JobClient: Running job: job_201204051216_0017 12/04/06 14:38:08 INFO mapred.JobClient: map 0% reduce 0% 12/04/06 14:38:26 INFO mapred.JobClient: Task Id : attempt_201204051216_0017_m_000005_0, Status : FAILED Error: class org.apache.lucene.analysis.ReusableAnalyzerBase overrides final method tokenStream.(Ljava/lang/String;Ljava/io/Reader;)Lorg/apache/lucene/analysis/TokenStream; 12/04/06 14:38:28 INFO mapred.JobClient: Task Id : attempt_201204051216_0017_m_000002_0, Status : FAILED Error: class org.apache.lucene.analysis.ReusableAnalyzerBase overrides final method tokenStream.(Ljava/lang/String;Ljava/io/Reader;)Lorg/apache/lucene/analysis/TokenStream; </code> From reading about this error, it appears that Lucene has been patched to make TokenStream final, resulting in the above issues. Mahout itself has an implementation of lucene jars that do not have this problem. I am assuming that the solution lies in getting the dse hadoop to use mahout lucene instead of the lucene it has. Any guidance on that? Cheers, Tristan