cassandra timeoutexception

(13 posts) (3 voices)

• Started 11 months ago by nare
• Latest reply from nare
• This topic is not resolved

Tags:

• data corruption
• timeoutexception

1. 

   nare
   Member

   Hello.

   I moved my cassandra from one sever to another. Cassandra's version on the first server was 1.0.8 on the new one it is 1.0.10. I moved also data by copying all files under /var/lib/cassandra/data/mykeyspace.
   and now it seems to me that some data has been corrupted during migration as I get the following exceptions when getting data for some keys.
   In cassandra client I get

   me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
   at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:42)
   at me.prettyprint.cassandra.service.KeyspaceServiceImpl$11.execute(KeyspaceServiceImpl.java:432)
   at me.prettyprint.cassandra.service.KeyspaceServiceImpl$11.execute(KeyspaceServiceImpl.java:416)
   at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)

   In cassandra system.log I get

   ERROR [ReadStage:492] 2012-07-02 21:50:50,280 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadStage:492,5,main]
   java.lang.AssertionError: 820255506 vs 518814720
   at org.apache.cassandra.io.util.MmappedSegmentedFile.floor(MmappedSegmentedFile.java:65)
   at org.apache.cassandra.io.util.MmappedSegmentedFile.getSegment(MmappedSegmentedFile.java:80)
   at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:830)

   Is this really data corruption problem and how can I fix it?

   Thanks.

   Posted 11 months ago #
2. 

   xedin
   Member

   Yes, this looks like corruption of the index segment, you can run ./bin/nodetool repair <ks> <cf> command to repair your data.

   Posted 11 months ago #
3. 

   nare
   Member

   I run the nodetool repair, it gave me no output. At least I didn't get any error but also it didn't help. I'm getting the same errors.

   Posted 11 months ago #
4. 

   xedin
   Member

   Wait, do you have another nodes in your ring to repair from?

   Posted 11 months ago #
5. 

   nare
   Member

   No just one node. The load on my server is low, so I don't need other nodes yet.
   And what if I delete all files from data directory except Data files?

   I have the following types of files

   MonitorResults-hc-18885-Data.db
   MonitorResults-hc-18885-Digest.sha1
   MonitorResults-hc-18885-Filter.db
   MonitorResults-hc-18885-Index.db
   MonitorResults-hc-18885-Statistics.db

   If I have keep only ...Data.db files and restart cassandra wouldn't other files be generated from this?

   Posted 11 months ago #
6. 

   nare
   Member

   I tried it but it didn't help. Without Digest, Index, .. etc files it didn't throw exceptions, but also it couldn't get data.

   Posted 11 months ago #
7. 

   xedin
   Member

   nare, unfortunately if you have only one node you have no way to repair all of the components you listed are inter-connected and represent a single SSTable so I don't see a way for you to repair your data set.

   Khan, could you be more precise about what you are saying?

   Posted 11 months ago #
8. 

   Khan
   Member

   I have also experienced the connection timeout exception when I moved from Apache Cassandra 1.0.9 to Apache Cassandra 1.1.1. I get an exception when I insert and read the data from Cassandra. There is no exception if I run the program in Apache Cassandra 1.0.9 but get connection timeout exception when I try to run it on Apache Casandra 1.1.1.

   Posted 11 months ago #
9. 

   nare
   Member

   I ran the nodetool scrub but it failed with the following exception.

   java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.EOFException: unable to seek to position 568688967 in /var/lib/cassandra/data/Monitis/MonitorResults-hc-18808-Data.db (518814720 bytes) in read-only mode
   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
   at java.util.concurrent.FutureTask.get(FutureTask.java:111)
   at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:204)
   ...

   Posted 11 months ago #
10. 

    xedin
    Member

    scrub wouldn't help you here because it runs on the existing sstables which are corrupted, you can also try running cleanup but I doubt it would change the situation.

    Posted 11 months ago #
11. 

    nare
    Member

    You were right. Cleanup also ended with exception

    Error occured during cleanup
    java.util.concurrent.ExecutionException: java.io.IOError: java.io.IOException: dataSize of 117171560 starting at 451517407 would be larger than file /var/lib/cassandra/data/Monitis/MonitorResults-hc-18808-Data.db length 518814720
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)

    Definitely my data or index file is corrupted and I can't fix it :(((

    Posted 11 months ago #
12. 

    xedin
    Member

    That is why people invented replication as a backup plan for situations like this :( The only way out for you would be to delete problematic sstable (all of it's components) you would loose a part of the data but system would be operational.

    Posted 11 months ago #
13. 

    nare
    Member

    Yes I already did it.
    Thanks for the help.

    Posted 11 months ago #

RSS feed for this topic