Hi,
we have (currently) a system where we are running Cassandra on two machines on a live setup. Every night the system takes automatic back-ups of the databases (Cassandra files) and then copies them to another set of ("development system") machines, which are running on identical set-up. Then the system stops the Cassandras running on dev system machines, copies the backups to said machines and starts the Cassandras again.
This has worked very well for a quite long time, but now we consistently see EOFException during the Cassandra start-up on the dev system machine. This happens every night when Cassandras are started. I'll paste the full exception log below. Has anyone encountered similar behavior / can we do something to fix this?
-J
ERROR [CompactionExecutor:4] 2013-02-04 01:11:20,442 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[CompactionExecutor:4,1,main]java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116) at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:99)
at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83)
at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:137)
at org.apache.cassandra.db.compaction.CompactionManager$2.runMayThrow(CompactionManager.java:164) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:112)
... 20 more