DataStax Support Forums » Topic: Data inserted via CQL does not get run through Solr's copyField directives

DataStax Support Forums » Topic: Data inserted via CQL does not get run through Solr's copyField directives http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives Software, Support, and Training for Apache Cassandra en-US Tue, 21 May 2013 09:18:47 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php jkusar on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-4322 Mon, 30 Jul 2012 21:24:55 +0000 jkusar 4322@http://www.datastax.com/support-forums/ Bah, I found my issue. I had some 2.0 binaries floating around in my path mucking things up since this didn't work in 2.0. Got everything cleaned up and it seems to be working now. Thanks! alexliu on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-4320 Mon, 30 Jul 2012 20:44:29 +0000 alexliu 4320@http://www.datastax.com/support-forums/ you can try test my example on DSE 2.1. I think it should work there as well. jkusar on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-4318 Mon, 30 Jul 2012 20:41:44 +0000 jkusar 4318@http://www.datastax.com/support-forums/ That's very interesting. How much longer till that version is released to the general public? Is it available as a beta anywhere? alexliu on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-4314 Mon, 30 Jul 2012 20:32:55 +0000 alexliu 4314@http://www.datastax.com/support-forums/ This is working in the incoming release, I am not sure DSE 2.1 the following is result that I run cqlsh> use ks33; cqlsh:ks33> insert into cf3 (key,title,body) values ('12345','This is a title','This is some body text'); cqlsh:ks33> select * from cf3; KEY | body | title -------+------------------------+----------------- 12345 | This is some body text | This is a title cqlsh:ks33> select * from cf3 where solr_query='text1:is'; KEY | body | title -------+------------------------+----------------- 12345 | This is some body text | This is a title cqlsh:ks33> select * from cf3 where solr_query='text1:body'; KEY | body | title -------+------------------------+----------------- 12345 | This is some body text | This is a title schema.xml <?xml version="1.0" encoding="UTF-8" ?> <schema name="wikipedia" version="1.1"> <types> <fieldType name="string" class="solr.StrField"/> <fieldType name="text" class="solr.TextField"> <analyzer><tokenizer class="solr.WikipediaTokenizerFactory"/></analyzer> </fieldType> </types> <fields> <field name="id" type="string" indexed="true" stored="true"/> <field name="name" type="text" indexed="true" stored="true"/> <field name="body" type="text" indexed="true" stored="true"/> <field name="title" type="text" indexed="true" stored="true"/> <field name="date" type="string" indexed="true" stored="true"/> <field name="text1" type="text" indexed="true" stored="true" multiValued="true"/> </fields> <defaultSearchField>body</defaultSearchField> <uniqueKey>id</uniqueKey> <copyField source="body" dest="text1"/> <copyField source="title" dest="text1"/> </schema> jkusar on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3897 Thu, 26 Jul 2012 18:07:23 +0000 jkusar 3897@http://www.datastax.com/support-forums/ Ok, here's some steps to reproduce the problem. First modify the wikipedia schema as follows: <pre><code><schema name="wikipedia" version="1.1"> <types> <fieldType name="string" class="solr.StrField"/> <fieldType name="text" class="solr.TextField"> <analyzer><tokenizer class="solr.WikipediaTokenizerFactory"/></analyzer> </fieldType> </types> <fields> <field name="id" type="string" indexed="true" stored="true"/> <field name="body" type="text" indexed="true" stored="true"/> <field name="text" type="text" indexed="true" stored="true" multiValued="true"/> <field name="title" type="text" indexed="true" stored="true"/> <field name="date" type="string" indexed="true" stored="true"/> </fields> <defaultSearchField>text</defaultSearchField> <uniqueKey>id</uniqueKey> <copyField source="body" dest="text"/> <copyField source="title" dest="text"/> </schema></code></pre> Run the add schema script. From cqlsh add a record: <code>cqlsh:wiki> insert into solr (key,title,body) values ('12345','This is a title','This is some body text');</code> Now from the solr admin tool, query for body. You will get nothing as it is searching the 'text' field and nothing has been copied to it. If you instead search for body:body (search for body within the body field), you get your record back. The copyField is definitely not happening. This is running a single solr node. The keyspace is using SimpleStrategy with RF=1. jkusar on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3896 Thu, 26 Jul 2012 17:47:40 +0000 jkusar 3896@http://www.datastax.com/support-forums/ I'm not using dynamic fields. All nodes are Solr. My schema is posted in another recent thread about copyField. tjake on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3884 Thu, 26 Jul 2012 15:01:35 +0000 tjake 3884@http://www.datastax.com/support-forums/ Are you using dynamic fields and are all your nodes Solr nodes or do you have a mix? jkusar on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3879 Thu, 26 Jul 2012 14:41:28 +0000 jkusar 3879@http://www.datastax.com/support-forums/ Hi Jake, I'm on 2.1. I realize it's not stored, but the admin app should still report the tokens that are indexed. One of my team members was able to find an instance where the fulltext search field has tokens in it. I'm trying to figure out why those got there when others didn't. There's definitely some weirdness going on as the simple app I put together does not copy the fields properly. Thanks. tjake on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3875 Thu, 26 Jul 2012 14:15:06 +0000 tjake 3875@http://www.datastax.com/support-forums/ jkusar what version of DSE are you on? 2.1 does support copyField from CQL. However the data in the copyField is not "stored" only indexed for search. jkusar on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3868 Thu, 26 Jul 2012 13:45:11 +0000 jkusar 3868@http://www.datastax.com/support-forums/ Xedin, are you a datastax employee? What you're saying is significantly different than what I've been told by other employees. Basically, I'm not supposed to have to bifurcate my data. I write to Cassandra and Solr gives me real time search capabilities and Hadoop gives me analytics. All for free (or so goes the datastax promise). xedin on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3840 Thu, 26 Jul 2012 09:30:55 +0000 xedin 3840@http://www.datastax.com/support-forums/ Yes, all the Solr specific things should be processed through Solr interface, all of the solr related indexes on the Cassandra CF could be viewed more like a Solr helpers for data persistence. jas on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3805 Thu, 26 Jul 2012 04:00:38 +0000 jas 3805@http://www.datastax.com/support-forums/ Okay, that's good to know. So, going through CQL, cassandra-cli, Hector etc., the indexes defined on the core column family are maintained, but any Solr-isms are outside its scope? The core definition files, including <code>schema.xml</code>, reside in the CF (placed there using the REST API), but mean nothing outside of Solr. How about the index-time textual analysis etc.? If there is Solr specific behavior required, then talk Solr to port 8983? I'm still talking to Solr, but I am looking into what I could do more directly using <code>sstableloader</code>. Maybe that's not such a good idea. Jeff xedin on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives/page/2#post-3776 Wed, 25 Jul 2012 20:48:04 +0000 xedin 3776@http://www.datastax.com/support-forums/ It's a feature as CQL is not supposed to process that kind of situation, as it doesn't know anything about Solr or how columns are interconnected using it. jkusar on "Data inserted via CQL does not get run through Solr's copyField directives" http://www.datastax.com/support-forums/topic/data-inserted-via-cql-does-not-get-run-through-solrs-copyfield-directives#post-3772 Wed, 25 Jul 2012 19:32:40 +0000 jkusar 3772@http://www.datastax.com/support-forums/ I've got a column family being indexed via solr. My Solr schema uses copyField to copy everything into a single text field so that I can do fulltext searching against it. I also use this, for example, to be able to have a field both tokenized as a text field and sortable as a string field. When I insert a record via the Solr interface this all works just great. However, if I insert a record via CQL, the columns themselves get indexed, but the copyField never happens meaning that fulltext searching doesn't work. My large text field is just empty. Is this a bug or a feature? I'm really hoping it's just a bug and that it will be fixed or else I have to scrap and re-write a huge chunk of code and operate almost exclusively via the Solr interface (far less than ideal). Assuming it is a bug, is there any chance it might be fixable in the near future? Thanks, Jason