Support Forums : DataStax

DataStax Support Forums » DataStax Enterprise

Data inserted via CQL does not get run through Solr's copyField directives

(14 posts) (5 voices)

Started 9 months ago by jkusar
Latest reply from jkusar
This topic is not resolved

Tags:

jkusar
Member

I've got a column family being indexed via solr. My Solr schema uses copyField to copy everything into a single text field so that I can do fulltext searching against it. I also use this, for example, to be able to have a field both tokenized as a text field and sortable as a string field.

When I insert a record via the Solr interface this all works just great. However, if I insert a record via CQL, the columns themselves get indexed, but the copyField never happens meaning that fulltext searching doesn't work. My large text field is just empty.

Is this a bug or a feature? I'm really hoping it's just a bug and that it will be fixed or else I have to scrap and re-write a huge chunk of code and operate almost exclusively via the Solr interface (far less than ideal). Assuming it is a bug, is there any chance it might be fixable in the near future?

Thanks,
Jason

Posted 9 months ago #
xedin
Member

It's a feature as CQL is not supposed to process that kind of situation, as it doesn't know anything about Solr or how columns are interconnected using it.

Posted 9 months ago #
jas
Member

Okay, that's good to know. So, going through CQL, cassandra-cli, Hector etc., the indexes defined on the core column family are maintained, but any Solr-isms are outside its scope? The core definition files, including schema.xml, reside in the CF (placed there using the REST API), but mean nothing outside of Solr.

How about the index-time textual analysis etc.? If there is Solr specific behavior required, then talk Solr to port 8983?

I'm still talking to Solr, but I am looking into what I could do more directly using sstableloader. Maybe that's not such a good idea.

Jeff

Posted 9 months ago #
xedin
Member

Yes, all the Solr specific things should be processed through Solr interface, all of the solr related indexes on the Cassandra CF could be viewed more like a Solr helpers for data persistence.

Posted 9 months ago #
jkusar
Member

Xedin, are you a datastax employee? What you're saying is significantly different than what I've been told by other employees. Basically, I'm not supposed to have to bifurcate my data. I write to Cassandra and Solr gives me real time search capabilities and Hadoop gives me analytics. All for free (or so goes the datastax promise).

Posted 9 months ago #
tjake
Moderator

jkusar what version of DSE are you on? 2.1 does support copyField from CQL. However the data in the copyField is not "stored" only indexed for search.

Posted 9 months ago #
jkusar
Member

Hi Jake,

I'm on 2.1. I realize it's not stored, but the admin app should still report the tokens that are indexed. One of my team members was able to find an instance where the fulltext search field has tokens in it. I'm trying to figure out why those got there when others didn't. There's definitely some weirdness going on as the simple app I put together does not copy the fields properly.

Thanks.

Posted 9 months ago #
tjake
Moderator

Are you using dynamic fields and are all your nodes Solr nodes or do you have a mix?

Posted 9 months ago #
jkusar
Member

I'm not using dynamic fields. All nodes are Solr. My schema is posted in another recent thread about copyField.

Posted 9 months ago #

jkusar
Member

Ok, here's some steps to reproduce the problem.

First modify the wikipedia schema as follows:

<schema name="wikipedia" version="1.1">
 <types>
  <fieldType name="string" class="solr.StrField"/>
  <fieldType name="text" class="solr.TextField">
    <analyzer><tokenizer class="solr.WikipediaTokenizerFactory"/></analyzer>
  </fieldType>
 </types>
 <fields>
    <field name="id"  type="string" indexed="true"  stored="true"/>
    <field name="body"  type="text" indexed="true"  stored="true"/>
    <field name="text"  type="text" indexed="true"  stored="true" multiValued="true"/>
    <field name="title"  type="text" indexed="true"  stored="true"/>
    <field name="date"  type="string" indexed="true"  stored="true"/>
 </fields>

<defaultSearchField>text</defaultSearchField>
<uniqueKey>id</uniqueKey>

<copyField source="body" dest="text"/>
<copyField source="title" dest="text"/>
</schema>

Run the add schema script.

From cqlsh add a record:
cqlsh:wiki> insert into solr (key,title,body) values ('12345','This is a title','This is some body text');

Now from the solr admin tool, query for body. You will get nothing as it is searching the 'text' field and nothing has been copied to it. If you instead search for body:body (search for body within the body field), you get your record back.

The copyField is definitely not happening.

This is running a single solr node. The keyspace is using SimpleStrategy with RF=1.

Posted 9 months ago #

alexliu
Member

This is working in the incoming release, I am not sure DSE 2.1

the following is result that I run

cqlsh> use ks33;

cqlsh:ks33> insert into cf3 (key,title,body) values ('12345','This is a title','This is some body text');
cqlsh:ks33> select * from cf3;
KEY | body | title
-------+------------------------+-----------------
12345 | This is some body text | This is a title

cqlsh:ks33> select * from cf3 where solr_query='text1:is';
KEY | body | title
-------+------------------------+-----------------
12345 | This is some body text | This is a title

cqlsh:ks33> select * from cf3 where solr_query='text1:body';
KEY | body | title
-------+------------------------+-----------------
12345 | This is some body text | This is a title

schema.xml

<?xml version="1.0" encoding="UTF-8" ?>
<schema name="wikipedia" version="1.1">
<types>
<fieldType name="string" class="solr.StrField"/>
<fieldType name="text" class="solr.TextField">
<analyzer><tokenizer class="solr.WikipediaTokenizerFactory"/></analyzer>
</fieldType>
</types>
<fields>
<field name="id" type="string" indexed="true" stored="true"/>
<field name="name" type="text" indexed="true" stored="true"/>
<field name="body" type="text" indexed="true" stored="true"/>
<field name="title" type="text" indexed="true" stored="true"/>
<field name="date" type="string" indexed="true" stored="true"/>
<field name="text1" type="text" indexed="true" stored="true" multiValued="true"/>
</fields>

<defaultSearchField>body</defaultSearchField>
<uniqueKey>id</uniqueKey>

<copyField source="body" dest="text1"/>
<copyField source="title" dest="text1"/>

</schema>

Posted 9 months ago #
jkusar
Member

That's very interesting. How much longer till that version is released to the general public? Is it available as a beta anywhere?

Posted 9 months ago #
alexliu
Member

you can try test my example on DSE 2.1. I think it should work there as well.

Posted 9 months ago #
jkusar
Member

Bah, I found my issue. I had some 2.0 binaries floating around in my path mucking things up since this didn't work in 2.0. Got everything cleaned up and it seems to be working now. Thanks!

Posted 9 months ago #

RSS feed for this topic

Reply

You must log in to post.