DataStax Support Forums » Tag: cache

DataStax Support Forums » Tag: cache - Recent Posts http://www.datastax.com/support-forums/tags/cache Software, Support, and Training for Apache Cassandra en-US Wed, 22 May 2013 02:13:23 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1830 Thu, 03 May 2012 15:44:20 +0000 jas 1830@http://www.datastax.com/support-forums/ Hi Rock: I had some things to take care of as well as I pursued the FVH problem using the more formal support ticket process. It turns out there was an FVH implementation bug in DSE 2.0 which has been reproduced and fixed and should be included in the DSE 2.1 release, whenever that is. I cannot speak for DataStax, of course, but I'm told the fix is in. :) That means I feel confident I can continue to develop around DSE 2.x and know that fix is coming. I look forward to when I can abandon plain old Solr completely, but until then, for high performance demos/testing, I'll continue to use Solr with FVH, and also work to incorporate DSE and Solr 4 features into my application. Cheers, Jeff Anonymous on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1805 Mon, 30 Apr 2012 20:47:00 +0000 Anonymous 1805@http://www.datastax.com/support-forums/ Jeff, To see the term vectors, this Solr component should be installed: <a href="http://wiki.apache.org/solr/TermVectorComponent" rel="nofollow">http://wiki.apache.org/solr/TermVectorComponent</a> Can you post your data and queries etc so we can reproduce the problem? It is likely a FVH bug. jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1794 Sat, 28 Apr 2012 20:14:45 +0000 jas 1794@http://www.datastax.com/support-forums/ Is it possible there is a bug in DSE 2 with regards to term offsets and positions when using FVH? I don't have a Solr 4.0 setup handy, but I did update my previous Solr 3.5.0 configuration to match that with DSE and I reindexed my content. Why the document ID is present in the DSE FVH snippet is a mystery. I do not see those in my Solr 3.5.0 output using FVH. It looks like the DSE non-FVH version posted previously, except there is a trailing space after each of the two closed <code></code> tag. So, why do I get document IDs appearing in the snippets using FVH with DSE 2? Looking at that particular document (ING:6xwoe) in Cassandra: <pre><code>[default@IngenuityContent] get SearchMain['ING:6xwoe']; ... => (column=n_name, value=egfr, timestamp=1335497830220000) ... => (column=n_synonym, value=egfr, timestamp=1335497830220000) ... => (column=type, value=node, timestamp=1335497830220000) Returned 24 results. Elapsed time: 22 msec(s). [default@IngenuityContent]</code></pre> Those field values do not contain the document ID. :) Is it possible to see the term offsets and positions somewhere? Thanks, Jeff jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1788 Fri, 27 Apr 2012 17:45:32 +0000 jas 1788@http://www.datastax.com/support-forums/ Anybody got any opinions regarding my posting from 4 days ago regarding how to best make use of 34GB of RAM using the heap, Solr caches, Cassandra row and key caches? I realize there is not specific magic answer, but a push in the right direction would be appreciated. Thanks, Jeff jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1785 Fri, 27 Apr 2012 16:30:40 +0000 jas 1785@http://www.datastax.com/support-forums/ I've not yet updated my EC2 instances, but I've been trying to resolve the FVH issue using my 2010 Core i7 iMac, with 16GB of RAM, using the DSE tarball install. A typical query generated by our application is used below, using the standard highlighter and then the fast vector highlighter. I'm using curl to speak directly to Solr/DSE, eliminating any additional application processing delays. The curl request is issued on the same system running DSE (localhost). <code>curl "http://localhost:8983/solr/IngenuityContent.SearchMain/select?hl.requireFieldMatch=true&facet=true&facet.offset=0&facet.mincount=1&facet.limit=8&hl=true&rows=10&fl=id,n_type,n_typeCategory,n_is_group,n_nameExact,n_synonymExact,n_macromolecule_name,n_macromolecule_id,n_macromolecule_species,n_macromolecule_summary,n_c_pubchem_cid,n_c_formula,n_c_cas_number,n_m_acc,n_m_descr&facet.sort=index&start=0&q=egfr&facet.field=n_diseaseFacet&facet.field=n_typeFacet&facet.field=n_tissue_typeFacet&facet.field=n_pathway_nameExact&facet.field=n_macromolecule_species&facet.field=n_locationFacet&facet.field=n_functionFacet&hl.usePhraseHighlighter=true&qt=partner-tmo&fq=type:node&debug=timing" > dse-search-egfr.timing.xml</code> The DSE log file shows: <pre><code>INFO [http-8983-3] 2012-04-27 10:09:39,469 SolrCore.java (line 1470) [IngenuityContent.SearchMain] webapp=/solr path=/select params={hl.requireFieldMatch=true&facet=true&facet.mincount=1&facet.offset=0&facet.limit=8&debug=timing&hl=true&rows=10&fl=id,n_type,n_typeCategory,n_is_group,n_nameExact,n_synonymExact,n_macromolecule_name,n_macromolecule_id,n_macromolecule_species,n_macromolecule_summary,n_c_pubchem_cid,n_c_formula,n_c_cas_number,n_m_acc,n_m_descr&facet.sort=index&start=0&q=egfr&facet.field=n_diseaseFacet&facet.field=n_typeFacet&facet.field=n_tissue_typeFacet&facet.field=n_pathway_nameExact&facet.field=n_macromolecule_species&facet.field=n_locationFacet&facet.field=n_functionFacet&qt=partner-tmo&hl.usePhraseHighlighter=true&fq=type:node} hits=2354 status=0 QTime=285</code></pre> Running this several times, the QTime varied from 279-292 ms. Well, the first time was 704 ms, but let's say that filled the caches... The Solr timing information for that request above, attributes 281 ms to <code>HighlightComponent</code>. Now, I added <code>hl.useFastVectorHighlighter=true</code> to the request: <code>curl "http://localhost:8983/solr/IngenuityContent.SearchMain/select?hl.requireFieldMatch=true&facet=true&facet.offset=0&facet.mincount=1&facet.limit=8&hl=true&hl.useFastVectorHighlighter=true&rows=10&fl=id,n_type,n_typeCategory,n_is_group,n_nameExact,n_synonymExact,n_macromolecule_name,n_macromolecule_id,n_macromolecule_species,n_macromolecule_summary,n_c_pubchem_cid,n_c_formula,n_c_cas_number,n_m_acc,n_m_descr&facet.sort=index&start=0&q=egfr&facet.field=n_diseaseFacet&facet.field=n_typeFacet&facet.field=n_tissue_typeFacet&facet.field=n_pathway_nameExact&facet.field=n_macromolecule_species&facet.field=n_locationFacet&facet.field=n_functionFacet&hl.usePhraseHighlighter=true&qt=partner-tmo&fq=type:node&debug=timing" > dse-search-egfr.timing.fast.xml</code> Log indicates: <pre><code>INFO [http-8983-3] 2012-04-27 10:15:48,687 SolrCore.java (line 1470) [IngenuityContent.SearchMain] webapp=/solr path=/select params={hl.requireFieldMatch=true&facet=true&facet.mincount=1&facet.offset=0&facet.limit=8&debug=timing&hl=true&rows=10&fl=id,n_type,n_typeCategory,n_is_group,n_nameExact,n_synonymExact,n_macromolecule_name,n_macromolecule_id,n_macromolecule_species,n_macromolecule_summary,n_c_pubchem_cid,n_c_formula,n_c_cas_number,n_m_acc,n_m_descr&hl.useFastVectorHighlighter=true&facet.sort=index&start=0&q=egfr&facet.field=n_diseaseFacet&facet.field=n_typeFacet&facet.field=n_tissue_typeFacet&facet.field=n_pathway_nameExact&facet.field=n_macromolecule_species&facet.field=n_locationFacet&facet.field=n_functionFacet&qt=partner-tmo&hl.usePhraseHighlighter=true&fq=type:node} hits=2354 status=0 QTime=182</code></pre> This was pretty consistent, varying from 181-192 ms. I will take a gain of 100 ms, thank you very much! :) The timing info reports 177 ms for <code>HighlightComponent</code>. This is admittedly a pretty narrow test case. But it shows I can get it to work and enjoy some benefits. Now, my problem is that the highlights produced by FVH are not yet ready for prime time. For example, for a given document, the old highlighter provides me: <pre><code><lst name="ING:6xwoe"> <arr name="n_name"> <str>egfr</str> </arr> <arr name="n_synonym"> <str>egfr</str> </arr> </lst></code></pre> For that same document, FVH gives: <pre><code><lst name="ING:6xwoe"> <arr name="n_name"> <str>ING:6xwoe egfr </str> </arr> <arr name="n_synonym"> <str>ING:6xwoe egfr </str> </arr> </lst></code></pre> For whatever reason, the document ID is included in the snippet, and FVH is highlighting a portion of it rather than egfr. Go figure... Oh well, that's what I have to do now. :) Cheers, Jeff jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1768 Wed, 25 Apr 2012 21:01:58 +0000 jas 1768@http://www.datastax.com/support-forums/ Rock: Hey, this is great! I will check out those links. Getting FVH to work is only one of several things i have to get done soon. :) Increasing the index size is not a problem. It's only a few gigs anyway, but highlighting is extremely important for the application. When issuing queries (to Solr 3.5.0 or DSE 2) with <code>debug=timing</code> it's apparent that highlighting is taking 90+% of the time typically. I've not yet gotten any responses from solr-users, but a good description of my issue can be found here: <a href="http://lucene.472066.n3.nabble.com/Re-FastVectorHighlighter-gt-no-highlights-td3933592.html" rel="nofollow">http://lucene.472066.n3.nabble.com/Re-FastVectorHighlighter-gt-no-highlights-td3933592.html</a> Cheers, Jeff Anonymous on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1767 Wed, 25 Apr 2012 18:53:09 +0000 Anonymous 1767@http://www.datastax.com/support-forums/ Also according to this link, multi term queries are not supported: <a href="http://grokbase.com/t/lucene/solr-user/1168fc837m/does-multiterm-highlighting-work-with-the-fastvectorhighlighter" rel="nofollow">http://grokbase.com/t/lucene/solr-user/1168fc837m/does-multiterm-highlighting-work-with-the-fastvectorhighlighter</a> Anonymous on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1766 Wed, 25 Apr 2012 18:44:06 +0000 Anonymous 1766@http://www.datastax.com/support-forums/ Jeff, Solr / Lucene is very fast for searching. The DSE / Cassandra interaction is minimal and so is not a factor in the highlighting speed. Highlighting is expensive because the entire document must be re-analyzed. Fast vector highlighter avoids this by utilizing term vectors. However, turning on (and storing) term vectors will increase the index size a lot. Sounds like FVH may be the best option for your situation, lets try to get it working for you. Can you post the solrconfig.xml you are using? This thread seems to illustrate the performance issues you are seeing with highlighting: <a href="http://lucene.472066.n3.nabble.com/Difference-between-regular-Highlighter-and-Fast-Vector-Highlighter-td2763162.html" rel="nofollow">http://lucene.472066.n3.nabble.com/Difference-between-regular-Highlighter-and-Fast-Vector-Highlighter-td2763162.html</a> jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance/page/2#post-1742 Sun, 22 Apr 2012 22:37:38 +0000 jas 1742@http://www.datastax.com/support-forums/ It's not too surprising that throwing more iron at the problem (moving from 2x m1.xlarge to 2x m2.2xlarge) helps with the situation, shaving about about 100ms. I still don't have FVH working yet, and I'm hoping that will also significantly improve my situation. With reserved pricing, it does not seem the m2.2xlarge cost will be too bad. While an m2.2xlarge also has four cores, this instance size has 13 ECUs vs 8 for the m1.xlarge, so each core is a lot more capable, thus the drop in response time. While the m1.xlarge has 15GB of RAM, the m2.x2large has 34GB. I still have my DSE heap size set to 8GB, and there is lots of free memory. How can I best take advantage of this memory? (1) Just generally create a much larger heap? Say 16GB, 24GB? Seems a GC could be quite a problem with a real huge heap. But, I see a lot of thought has been put into configuring GC related parameters in cassandra-env.sh, implying that GC pauses may not be too bad. (2) Given this large heap, and that my application is very Solr query centric (reads), should I just enlarge the Solr document, filter, query result and field value caches to keep more content in memory, making it more readily accessible to Solr? I've only currently got maybe 5GB indexed in Solr, so in theory I could put it all in memory. That could result in quite a lot of tenured objects. (3) Are there Cassandra specific settings that can take more advantage of the available memory? Does it make sense to increase the size of the row and key caches? Or, with (2) in place, does that matter given my Solr centricity? Thanks, Jeff jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance#post-1733 Fri, 20 Apr 2012 18:33:39 +0000 jas 1733@http://www.datastax.com/support-forums/ Hi Rock: Highlighting is my pain point still. It looks like Solr 3.5.0 can do the highlighting a little faster than DSE 2. I've not tried a pure Solr 4.0 to know if there's simply slower highlighting there, or if it's a DSE thing. One improvement I made was to change my edismax config from <code>hl.fl=*</code> to just itemizing the 50 some fields I'm actually searching on (<code>qf</code>), and that helped a little. No point highlighting where its' not needed. For whatever reason, I thought <code>*</code> meant to highlight all in <code>qf</code>, but it became apparent it was all fields. I'm trying to get the Solr FastVectorHighlighter to work, and that requires additional field attributes in <code>schema.xml</code>. I noticed in the DSE log for several fields that since those things were not present, it was reverting to the old highlighter. That pointed out the fact I was highlighting more fields than I wanted. Anyway, I tried various cache and other configurations in <code>solrconfig.xml</code>, but the real problem is highlighting, even for a single request. The system does not have to be under load to perform poorly. Of course, the more rows I have returned, the worse it gets. Using a couple of Solr books I have, and many pages on the web, I've still not be able to get FastVectorHighlighter (FVH) to work for me. It's much faster, but I'm not getting highlights. :) I get a <code><highlighting></code> section, with entries for the expected document IDs, but no highlights. I sent email to the solr-users email list yesterday morning, but no one has addressed it. Apparently FVH has issues with ngram'd fields (if min size != max size) and multi-term (not sure what that is). I'm not using those as far as I know, but most of my fields are shingled. I hope I can figure this out and take some noticeable chunk off of highlighting time. Cheers, Jeff Anonymous on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance#post-1730 Fri, 20 Apr 2012 15:01:50 +0000 Anonymous 1730@http://www.datastax.com/support-forums/ Jeff, Are you still seeing performance issues? The main one you mentioned seemed to essentially revolve around highlighting? Rock jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance#post-1717 Thu, 19 Apr 2012 18:36:12 +0000 jas 1717@http://www.datastax.com/support-forums/ H Jake: I'm using only a single instance of Solr 3.5.0. We did have it set up master/slave a while back, but that was to load balance queries. The slave complete replicated the master. There was no sharding. Does <code>cl=xx</code> apply to queries as well as updates with DSE? Thanks, Jeff tjake on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance#post-1711 Thu, 19 Apr 2012 17:19:48 +0000 tjake 1711@http://www.datastax.com/support-forums/ Is your solr 3.5 setup using 2 shards as well? jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance#post-1668 Tue, 17 Apr 2012 22:00:06 +0000 jas 1668@http://www.datastax.com/support-forums/ Ah, I see the default consistency level is QUORUM. The docs illustrate using the <code>cl</code> parameter: <pre><code><a href="http://<host>:<port>/solr/<keyspace>.<column" rel="nofollow">http://<host>:<port>/solr/<keyspace>.<column</a> family>/update?cl=ONE</code></pre> Does this only apply to updates, or can I specify <code>cl=ONE</code> on a query as well? Guess I'll try it out. Jeff jas on "Levers for increasing DSE 2 Solr performance?" http://www.datastax.com/support-forums/topic/levers-for-increasing-dse-2-solr-performance#post-1667 Tue, 17 Apr 2012 21:46:14 +0000 jas 1667@http://www.datastax.com/support-forums/ I'm currently looking into the FastVectorHighlighter. But, I got a little distracted and noticed in the DSE 2.0 documentation, the section "Increasing Read Performance by Adding Replicas". How does that work to increase performance? I do have 2 nodes, and replication configured: <pre><code>[default@unknown] describe IngenuityContent; Keyspace: IngenuityContent: Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy Durable Writes: true Options: [Solr:2] Column Families: ColumnFamily: ...</code></pre> I interact with Solr/DSE using SolrJ via the default port 8983. With two nodes, I have a load balancer between my application instances and DSE for search. For regular Cassandra access I use Hector and configure it with the two nodes. My understanding is that it does software based load balancing in issuing requests to one coordinator node or another. When my application issues a Solr query to a given DSE node, how does having a replica increase its performance? I assume the node the app (or load balancer) selected does all of the Solr work, right? Does the replica participate in providing some of the underlying data to the coordinator node? Is this benefit only achieved if I 'query Solr' via CQL rather than SolrJ? Oh, at what consistency level does DSE work with for Solr queries? The particular core that is giving me this performance issue contains canonical content which is updated by my company only once a <ul>quarter</ul> . There are other cores subject to updates initiated at any time by tenants, but the canonical content can easily work with consistency level ONE. Is there a way to instruct DSE to do that and avoid getting data from both replicas etc.? Thanks, Jeff