Support Forums : DataStax

DataStax Support Forums » DataStax Enterprise

How does Solr re-indexing get triggered?

(18 posts) (3 voices)

Started 1 year ago by
Latest reply from
This topic is resolved

Tags:

12 Next »

jas
Member

What is it that triggers DSE 2 to re-index? That is, I have created a core using the HTTP APIs and POSTed solrconfig.xml, schema.xml etc. I then POSTed Solr XML formatted content to the Solr port (8983) of DSE 2. I can see the keyspace and column family exist, and I can see the config files, as well as rows in the CF keyed by the document IDs. If I describe the keyspace, I can see all of the fields defined as columns, configured with a CUSTOM secondary index etc.

I'm currently making changes to both solrconfig.xml and schema.xml in order to work with the Solr FactVectorHighlighter to hopefully improve highlighting performance. In the past, if I changed schema.xml, I'd have to feed all of the content to Solr again. How does this work with DSE? When I post either file there is quite a bit stuff logged regarding applying a migration an updating the column family, as well as outputting the fields defined in schema.xml.

The DSE docs describes how data can be inserted via Cassandra or Solr. But, with data already in place, when I modify schema.xml, how do I re-index that existing data? Is it automatic or do I have to trigger it in some way?

Thanks,

Jeff

Posted 1 year ago #
tjake
Moderator

When you post the new schema.xml it will automatically start rebuilding the index from the source data in the CF

Posted 1 year ago #
jas
Member

Thanks Jake. Does that only apply to schema.xml then? If, say, I add some synonyms to synonyms.txt and POST that, does that trigger indexing, or would I have to followup with an unmodified schema.xml?

Posted 1 year ago #
tjake
Moderator

Any file will kick off the rebuild. you can track progress with nodetool compactionstats

Posted 1 year ago #
jas
Member

Great. Thanks for the usual fast response! :)

Jeff

Posted 1 year ago #

jas
Member

Okay, I'm starting to have my doubts... First off, I removed a field from qf in my edismax configuration but the field is still being returned when I search. And, I modified my schema to make use of the fast vector highlighter. For example:

<field name="n_cellreg_role_in_cell" type="text_lc_np_shingle" indexed="true" stored="true" multiValued="true" termVectors="true" termPositions="true" termOffsets="true" />

For that field and the others I modified, I see in system.log:

WARN [http-8983-5] 2012-04-18 14:43:08,731 DefaultSolrHighlighter.java (line 421) Solr will use Highlighter instead of FastVectorHighlighter because n_cellreg_role_in_cell field does not store TermPositions and TermOffsets.
 WARN [http-8983-5] 2012-04-18 14:43:08,731 DefaultSolrHighlighter.java (line 421) Solr will use Highlighter instead of FastVectorHighlighter because n_member_nameExact field does not store TermPositions and TermOffsets.

Using nodetool compactionstats, shows 0 pending tasks.

Tracing my core definition/update script when posting these two files:

[imac:solr/multicore/ing-content] jas% ./define-core.sh
+ CORENAME=IngenuityContent.SearchMain
+ SOLRCORE_URL=http://localhost:8983/solr/resource/IngenuityContent.SearchMain
+ echo 'Posting core configuration files to: http://localhost:8983/solr/resource/IngenuityContent.SearchMain&#039;
Posting core configuration files to: http://localhost:8983/solr/resource/IngenuityContent.SearchMain
+ curl http://localhost:8983/solr/resource/IngenuityContent.SearchMain/solrconfig.xml --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'
+ curl http://localhost:8983/solr/resource/IngenuityContent.SearchMain/schema.xml --data-binary @schema.xml -H 'Content-type:text/xml; charset=utf-8'
+ echo 'Created/updated core: IngenuityContent.SearchMain'
Created/updated core: IngenuityContent.SearchMain
[imac:solr/multicore/ing-content] jas%

Thanks,

Jeff

Posted 1 year ago #

jas
Member

FYI, if I describe the keyspace, I can see the latest solrconfig.xml and schema.xml are present. It just seems like they've not been applied.

Jeff

Posted 1 year ago #
jas
Member
Interesting, out of desperation, I ran dsetool rebuild_indexes .... That took several minutes to run. When it was done, I noticed no difference. So I then killed DSE, let it terminate gracefully, and then started it again.

Then I saw this in the log when I issued a query:
```
java.lang.RuntimeException: SchemaField: n_member_type conflicting 'true' field options for non-indexed field:{termPositions=true, multiValued=true, indexed=false, stored=true, termVectors=true, termOffsets=true}
	at org.apache.solr.schema.SchemaField.calcProps(SchemaField.java:228)
	at org.apache.solr.schema.SchemaField.create(SchemaField.java:188)
```
Yes! That field was mis-configured since it's not being indexed, just stored for retrieval. I fixed that, POSTed schema.xml again and got the same error when querying. I killed and restarted DSE, and that error no longer appeared, and it looks like the Fast Vector Highlighter is being used on at least some of the fields. I need to look into that further.

Cheers,

Jeff
Posted 1 year ago #
tjake
Moderator

Hi Jeff,

It sounds like we didn't catch the error upfront when you posted the new schema. I'll look at this and see if it can be thrown upfront.

-Jake

Posted 1 year ago #
jas
Member

Hi Jake:

I can see that re-indexing does occur as long as the config files are valid. Will 'thrown upfront' entail an error coming back via the HTTP API in addition to the log?

Jeff

Posted 1 year ago #
tjake
Moderator

Yes that's what it would be.

Posted 1 year ago #
jas
Member

Trying to resolve my highlighting related performance issue in http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance, I figured I'd explicitly re-index my content (POST XML to Solr) and see what happens. As I mentioned earlier, once I got schema.xml w/o errors and no longer saw log messages indicating Solr was reverting to the old highlighter, I assumed the actual term positions and offsets were actually generated based on the content already in Cassandra.

I guess that was not the case though. After finding nothing directly applicable to my issue regarding FVH on the web, I decided what the heck, re-index all of the content by posting my existing Solr XML files. Well, I'm getting highlights now with FVH! Unfortunately, they are not lined up properly, but I hope that's something goofy I'm doing that I can more readily resolve.

Posted 1 year ago #
tjake
Moderator

So even when rebuilding the index with dsetool rebuild_index command you saw no affect?

What kind of performance are you seeing now?

Posted 1 year ago #
jas
Member

Hi Jake:

I tried dsetool a week ago as a desperate attempt. I noticed in the docs that rebuilding indexes using dsetool is presented in the guise of a corrupted index. One takes the node offline, shuts it down, deletes the index directory, restarts, and then uses dsetool to rebuild the index.

Is it possible to force reindexing w/o going through all that? I figured it was too good to be true to simply POST a new version of schema.xml and have the existing content reindexed. That obviously won't work for new fields, but not all fields are actually stored (just indexed), and could not be re-indexed with new analysis configurations, adding term offsets and positions etc.

I'll be back with some performance data.

Jeff

Posted 1 year ago #
tjake
Moderator

Yes you can rebuild the index without removing the index. It will rebuild live.

Posted 1 year ago #

RSS feed for this topic

12 Next »

Reply »

You must log in to post.