http://www.datastax.com/support-forums/topic/fatal-exception-while-updating-core-configuraton Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 05:17:04 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/fatal-exception-while-updating-core-configuraton#post-1736 Fri, 20 Apr 2012 18:57:06 +0000 jas 1736@http://www.datastax.com/support-forums/ <p>No problem Jake. I figured it was something along those lines. :) </p> http://www.datastax.com/support-forums/topic/fatal-exception-while-updating-core-configuraton#post-1735 Fri, 20 Apr 2012 18:53:45 +0000 tjake 1735@http://www.datastax.com/support-forums/ <p>The stack trace indicates this is related to the heap pressure detection thread which checks the size of the cassandra memtables and also the solr in memory size. It must have kicked off during the core reload. This is a minor thing, but I will file a ticket to make sure we don't throw this in this case.</p> <p>Thanks for the info.</p> <p>-Jake </p> http://www.datastax.com/support-forums/topic/fatal-exception-while-updating-core-configuraton#post-1734 Fri, 20 Apr 2012 18:47:33 +0000 jas 1734@http://www.datastax.com/support-forums/ <p>Hi:</p> <p>I figured I'd share this with you FYI. I've not seen this happen before and I've POSTed several revisions to <code>solrconfig.xml</code> and <code>schema.xml</code>. However, I just launched an instance of the single node DSE AMI I created a while back started DSE and issued some queries. Then I wanted to apply the core updates I've made on other nodes and this happened.</p> <p>I applied the updates again, and no errors occurred that time, so I'm proceeding. But, it seems operating on a closed <code>IndexWriter</code> is not a good thing and I wanted to at least provide some of the log messages and the stack trace to make you aware:</p> <pre><code>INFO [http-8983-1] 2012-04-20 18:13:19,952 RequestHandlers.java (line 152) adding lazy requestHandler: solr.SearchHandler INFO [http-8983-1] 2012-04-20 18:13:19,952 RequestHandlers.java (line 170) created /terms: solr.SearchHandler INFO [http-8983-1] 2012-04-20 18:13:20,025 SolrIndexSearcher.java (line 119) reader:CassandraDirectoryReader(segments_f:1333551713915 _2c(4.0):Cv13130/5794 _c(4.0):Cv22471 _p(4.0):Cv22721 _16(4.0):Cv19296 _1r(4.0):Cv18508 _36(4.0):Cv750 _1h(4.0):Cv16118 _22(4.0):Cv18366 _2n(4.0):Cv20339/20339 _3e(4.0):Cv2446/299 _3z(4.0):Cv3157 _4i(4.0):Cv3131 _3o(4.0):Cv2630 _48(4.0):Cv3343 _4u(4.0):Cv2970 _55(4.0):Cv2994 _4v(4.0):Cv350 _5m(4.0):Cv2779 _54(4.0):Cv325 _58(4.0):Cv337 _56(4.0):Cv354 _59(4.0):Cv209 _5c(4.0):Cv393 _5f(4.0):Cv273 _5d(4.0):Cv340 _5i(4.0):Cv404 _5g(4.0):Cv276 _5h(4.0):Cv369 _5j(4.0):Cv290 _5k(4.0):Cv268 _5l(4.0):Cv179 _5q(4.0):Cv182 _5n(4.0):Cv285 _5s(4.0):Cv43 _5t(4.0):Cv5 _5p(4.0):Cv195 _5r(4.0):Cv170 _5o(4.0):Cv146) SolrAssertions.searcherReader:null INFO [http-8983-1] 2012-04-20 18:13:20,026 SolrIndexSearcher.java (line 128) Opening Searcher@5afe1192 main INFO [http-8983-1] 2012-04-20 18:13:20,029 CommitTracker.java (line 79) Hard AutoCommit: disabled INFO [http-8983-1] 2012-04-20 18:13:20,029 CommitTracker.java (line 79) Soft AutoCommit: if uncommited for 1000ms; INFO [http-8983-1] 2012-04-20 18:13:20,029 CassandraDirectUpdateHandler2.java (line 89) Created new solr core: IngenuityContent.SearchMain INFO [http-8983-1] 2012-04-20 18:13:20,030 SpellCheckComponent.java (line 589) Initializing spell checkers WARN [http-8983-1] 2012-04-20 18:13:20,031 SpellCheckComponent.java (line 638) No queryConverter defined, using default converter INFO [http-8983-1] 2012-04-20 18:13:20,032 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.QueryComponent@4b465dd5 INFO [http-8983-1] 2012-04-20 18:13:20,032 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.FacetComponent@78ba718e INFO [http-8983-1] 2012-04-20 18:13:20,032 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.MoreLikeThisComponent@18755f8e INFO [http-8983-1] 2012-04-20 18:13:20,033 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.HighlightComponent@41d4702d INFO [http-8983-1] 2012-04-20 18:13:20,033 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.StatsComponent@6210bc1e INFO [http-8983-1] 2012-04-20 18:13:20,033 SearchHandler.java (line 133) Adding debug component:org.apache.solr.handler.component.DebugComponent@38ae87de INFO [http-8983-1] 2012-04-20 18:13:20,033 SearchHandler.java (line 128) Adding component:com.datastax.bdp.cassandra.index.solr.CassandraQueryComponent@621ee2de INFO [http-8983-1] 2012-04-20 18:13:20,033 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.QueryComponent@4b465dd5 INFO [http-8983-1] 2012-04-20 18:13:20,033 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.FacetComponent@78ba718e INFO [http-8983-1] 2012-04-20 18:13:20,033 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.MoreLikeThisComponent@18755f8e INFO [http-8983-1] 2012-04-20 18:13:20,034 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.HighlightComponent@41d4702d INFO [http-8983-1] 2012-04-20 18:13:20,034 SearchHandler.java (line 128) Adding component:org.apache.solr.handler.component.StatsComponent@6210bc1e INFO [http-8983-1] 2012-04-20 18:13:20,034 SearchHandler.java (line 133) Adding debug component:org.apache.solr.handler.component.DebugComponent@38ae87de INFO [pool-40-thread-1] 2012-04-20 18:13:20,037 QuerySenderListener.java (line 43) QuerySenderListener sending requests to Searcher@5afe1192 main INFO [pool-40-thread-1] 2012-04-20 18:13:20,039 SolrCore.java (line 1470) [IngenuityContent.SearchMain] webapp=null path=null params={event=firstSearcher&amp;q=brca1} hits=0 status=0 QTime=1 INFO [pool-40-thread-1] 2012-04-20 18:13:20,039 QuerySenderListener.java (line 83) QuerySenderListener done. INFO [pool-40-thread-1] 2012-04-20 18:13:20,039 SpellCheckComponent.java (line 677) Loading spell index for spellchecker: default INFO [http-8983-1] 2012-04-20 18:13:20,040 CoreContainer.java (line 537) replacing core: IngenuityContent.SearchMain INFO [http-8983-1] 2012-04-20 18:13:20,040 SolrCore.java (line 733) [IngenuityContent.SearchMain] CLOSING SolrCore org.apache.solr.core.SolrCore@3668ad91 INFO [pool-40-thread-1] 2012-04-20 18:13:20,041 SolrCore.java (line 1412) [IngenuityContent.SearchMain] Registered new searcher Searcher@5afe1192 main INFO [http-8983-1] 2012-04-20 18:13:20,043 SolrCore.java (line 1428) [IngenuityContent.SearchMain] Closing main searcher on request. INFO [http-8983-1] 2012-04-20 18:13:20,043 SolrIndexSearcher.java (line 231) Closing Searcher@29defa49 main fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} INFO [http-8983-1] 2012-04-20 18:13:20,062 DirectUpdateHandler2.java (line 446) closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommit maxTime=1000ms,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} INFO [OptionalTasks:1] 2012-04-20 18:13:20,590 SolrDeletionPolicy.java (line 114) SolrDeletionPolicy.onInit: commits:num=1 commit{dir=/usr/local/ingenuity/isec/dse/cassandra/datastore/data/solr.data/IngenuityContent.SearchMain/index,segFN=segments_f,version=1333551713915,generation=15,filenames=[_4i.fdt, _59.tvd, _2n.fdx, _59.tvf, _5l.nrm, _4u_0.frq, _2n.fdt, _5r_0.frq, _3z_0.frq, _5i_0.frq, _5j.tvd, _5q.tvx, _5j.tvf, _5r.fdt, _56.tvx, _5r.fdx, _3o.tvf, _59_0.frq, _3o.tvd, _5j.tvx, _54.tvx, _5f.fdx, _5p_0.frq, _55_0.tip, _55_0.tim, _5f.fdt, _5k_0.tim, _5o_0.prx, _5t_0.frq, _4i.fdx, _1r_0.prx, _5k_0.tip, _3z_0.tip, _59.fnm, _3z_0.tim, _5q.tvd, _3o.nrm, _5q.tvf, _56_0.frq, _5c.fdt, _5c.fdx, _54.nrm, _3e.fdx, _4i.tvx, _3z.nrm, _3e.fdt, _5j.fnm, _1h_0.prx, _5m.fdx, _22_0.tim, _4i.tvf, _22_0.tip, _4i.tvd, _22.fdt, _2c_0.prx, _5g_0.tim, _22.fdx, _5g_0.tip, _5l.fnm, _5m.fdt, _5j.nrm, _16.fdx, _c.fdx, _5l.tvx, _c.fdt, _p.nrm, _16.fdt, _5l.tvd, _3e_0.frq, _5l.tvf, _36_0.prx, _2c_0.frq, _54_0.prx, _59.nrm, _5h.fdt, _4u_0.prx, _5h.fdx, _5h_0.tip, _5h_0.tim, _58.nrm, _56_0.tim, _56_0.tip, _3o.fnm, _36.fdx, _36.fdt, _3e.fnm, _5g.nrm, _59_0.tim, _54.tvf, _c.tvx, _54.tvd, _4u.fdx, _48.fdx, _3e_0.prx, _4u.fdt, _48.fdt, _5k.nrm, _56.tvd, _56.tvf, _2c_0.tip, _5c_0.tim, _5c_0.tip, _2c_0.tim, _5r_0.tim, _c.tvf, _48_0.prx, _5r_0.tip, _c.tvd, _5n_0.tim, _5s_0.frq, _5n_0.tip, _59_0.tip, _5q_0.tim, _5r.fnm, _5q_0.tip, _2c.fdx, _2c.fdt, _5p_0.prx, _c.nrm, _2n_0.frq, _22.tvx, _5o.tvf, segments_f, _5o.tvd, _4i_0.tim, _22.tvd, _22.tvf, _5k.fnm, _2n_0.tim, _5s.fdt, _2n_0.tip, _4i_0.tip, _5s.fdx, _5r.tvx, _5d.fdx, _5n_0.prx, _5g_0.prx, _5o.tvx, _5p.fdt, _16.tvd, _3e_1.del, _16.tvf, _5r.tvf, _5p.fdx, _5m.fnm, _1h.nrm, _5r.tvd, _5d.fdt, _4v.fdt, _58.tvf, _4v.fdx, _p_0.tip, _5j.fdx, _p_0.tim, _5t.fnm, _48.nrm, _58.tvd, _5o_0.tim, _56.fnm, _5o_0.tip, _59.fdt, _4v_0.tim, _4v_0.tip, _59.fdx, _56.fdt, _2n.fnm, _56.fdx, _5m_0.frq, _c.fnm, _5i_0.prx, _2n.nrm, _3e.tvx, _5o.fdt, _5o.fdx, _5l_0.tip, _5j_0.prx, _5l_0.tim, _4u.tvx, _5j_0.frq, _3o.fdt, _5q_0.frq, _3o.fdx, _3e.tvf, _3e.tvd, _36.fnm, _3o_0.frq, _59_0.prx, _4u.tvd, _4u.tvf, _58.tvx, _5n_0.frq, _48.fnm, _5q.nrm, _5s_0.tim, _5s_0.tip, _1r.nrm, _3o.tvx, _2n.tvd, _c_0.prx, _5d_0.frq, _1h.tvx, _16_0.tim, _5d_0.prx, _1h.tvd, _2n.tvf, _1h.tvf, _59.tvx, _5h.tvf, _5r.nrm, _5h.tvd, _2n.tvx, _16_0.tip, 2.fnx, _5h.tvx, _5f.tvd, _5f.tvf, _5t.nrm, _5p.fnm, _5h_0.prx, _5n.tvd, _5n.tvf, _5t.tvd, _5f.fnm, _4i.nrm, _4i_0.frq, _5t.tvf, _2c_1.del, _5f.tvx, _5q_0.prx, _5k.fdx, _22_0.frq, _5s.fnm, _5p_0.tip, _5p_0.tim, _1h.fdx, _4v.fnm, _36.nrm, _4i_0.prx, _2n_2.del, _c_0.frq, _22.nrm, _5g.fdx, _5t.fdx, _5c_0.frq, _5g.fdt, _3z.tvx, _55.tvx, _5t.fdt, _5t.tvx, _4u.nrm, _5n.tvx, _3z.fdt, _3z.fdx, _22.fnm, _5g_0.frq, _5h_0.frq, _55.fdt, _5l.fdt, _5k_0.frq, _p.fdx, _5g.fnm, _2c.tvx, _c_0.tim, _p.fdt, _5m.nrm, _5h.fnm, _55.fdx, _5j.fdt, _5p.tvx, _2c.tvf, _2c.tvd, _5m_0.prx, _3z.tvd, _2c.fnm, _3z.tvf, _54.fdx, _5p.nrm, _5m.tvx, _54.fdt, _1h_0.tip, _5j_0.tip, _5f.nrm, _1h_0.tim, _5m.tvd, _5n.nrm, _55_0.frq, _5i.nrm, _3o_0.prx, _5m.tvf, _5j_0.tim, _5h.nrm, _5n.fdt, _5n.fdx, _58_0.frq, _4v_0.prx, _5t_0.tim, _58_0.tip, _5t_0.tip, _5o.fnm, _1h.fdt, _5d.tvf, _5d.tvd, _16.tvx, _5d.tvx, _5d.nrm, _5k.fdt, _4v_0.frq, _5p.tvf, _58.fnm, _1r_0.tim, _1r_0.tip, _5q.fnm, _5i.fnm, _16_0.prx, _5o.nrm, _5d_0.tip, _58_0.tim, _5p.tvd, _c_0.tip, _4i.fnm, _5d_0.tim, _5k_0.prx, _1h.fnm, _5g.tvd, _48.tvx, _5g.tvf, _5g.tvx, _3z_0.prx, _56.nrm, _55_0.prx, _55.nrm, _5f_0.frq, _36.tvf, _3e_0.tip, _36.tvd, _3e_0.tim, _5t_0.prx, _48.tvd, _5i.fdx, _48.tvf, _1r.fdx, _5i.fdt, _1r.fdt, _5i.tvx, _58_0.prx, _3z.fnm, _5i.tvf, _16_0.frq, _36.tvx, _5l_0.prx, _36_0.tip, _5f_0.tim, _5f_0.tip, _5i.tvd, _4v.nrm, _5l.fdx, _36_0.tim, _5c.tvf, _5c.tvd, _36_0.frq, _5s_0.prx, _16.nrm, _5c.tvx, _5o_0.frq, _5l_0.frq, _3o_0.tip, _1r_0.frq, _5s.tvx, _1r.tvf, _5n.fnm, _1r.tvd, _5k.tvf, _54_0.tip, _54.fnm, _5k.tvd, _2c.nrm, _5c_0.prx, _3o_0.tim, _1r.fnm, _54_0.tim, _16.fnm, _56_0.prx, _48_0.tim, _5r_0.prx, _22_0.prx, _5k.tvx, _55.fnm, _5i_0.tim, _p_0.prx, _p_0.frq, _2n_0.prx, _5m_0.tim, _5f_0.prx, _55.tvf, _55.tvd, _5d.fnm, _5m_0.tip, _54_0.frq, _5c.fnm, _5i_0.tip, _58.fdt, _5c.nrm, _5s.nrm, _p.fnm, _p.tvx, _58.fdx, _1h_0.frq, _4v.tvf, _4u_0.tip, _4v.tvd, _5q.fdx, _4u_0.tim, _5q.fdt, _p.tvd, _p.tvf, _48_0.tip, _48_0.frq, _3e.nrm, _5s.tvd, _5s.tvf, _4v.tvx, _4u.fnm, _1r.tvx] INFO [OptionalTasks:1] 2012-04-20 18:13:20,591 SolrDeletionPolicy.java (line 136) newest commit = 1333551713915 ERROR [OptionalTasks:1] 2012-04-20 18:13:20,605 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[OptionalTasks:1,5,main] org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:792) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:797) at org.apache.lucene.index.IndexWriter.ramSizeInBytes(IndexWriter.java:3063) at com.datastax.bdp.cassandra.index.solr.CassandraDirectUpdateHandler2.getLiveSize(CassandraDirectUpdateHandler2.java:786) at com.datastax.bdp.cassandra.index.solr.SolrSecondaryIndex.getLiveSize(SolrSecondaryIndex.java:422) at org.apache.cassandra.db.index.SecondaryIndexManager.getTotalLiveSize(SecondaryIndexManager.java:361) at org.apache.cassandra.db.ColumnFamilyStore.getTotalMemtableLiveSize(ColumnFamilyStore.java:1032) at org.apache.cassandra.db.MeteredFlusher.run(MeteredFlusher.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) INFO [http-8983-1] 2012-04-20 18:13:20,606 DirectUpdateHandler2.java (line 455) closed DirectUpdateHandler2{commits=0,autocommits=0,soft autocommit maxTime=1000ms,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}</code></pre>