Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

Cassandra 1.1 bails at startup with NP Exception after schema change; how to fix?

(6 posts) (2 voices)

Started 10 months ago by ioncube
Latest reply from xedin
This topic is not a support question

Tags:

No tags yet.

ioncube
Member

Using Cassandra 1.1.0 on Linux, Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode).

There may be several issues here, including user error most likely, but the result is that trying to start Cassandra now bails with a null pointer exception.

Background:

1) We found that phpcassa cannot correctly return data from the dummy 'timeline' table, created thus, and have been exploring why:

CREATE TABLE timeline (
user_id varchar,
tweet_id uuid,
author varchar,
body varchar,
PRIMARY KEY (user_id, tweet_id)
);

2) A 'show schema' showed that there's no column metadata created in a CF with compound primary - maybe that's correct or perhaps a CQL3 issue, we're not sure.

3) The key validation class was also not compound - probably correct but not sure.

4) Decided to try altering the key_validation_class as follows and the result was a transport exception after Cassandra blew up and now a null pointer exceptions at startup.

[default@demodb] update column family timeline with key_validation_class = 'CompositeType(UUIDType,UTF8Type)';
org.apache.thrift.transport.TTransportException

5) So we presumably did something dumb to the schema, but these things could potentially happen in a live setting. How can we recover without significant dataloss and correct the schema if necessary so that the server can start?

The startup backtrace is:

/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17 (346 bytes)
ERROR 17:01:11,060 Exception encountered during startup
java.lang.NullPointerException
at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:163)
at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:120)
at org.apache.cassandra.cql.jdbc.JdbcUTF8.getString(JdbcUTF8.java:77)
at org.apache.cassandra.cql.jdbc.JdbcUTF8.compose(JdbcUTF8.java:97)
at org.apache.cassandra.db.marshal.UTF8Type.compose(UTF8Type.java:35)
at org.apache.cassandra.cql3.UntypedResultSet$Row.getString(UntypedResultSet.java:87)
at org.apache.cassandra.config.ColumnDefinition.fromSchema(ColumnDefinition.java:256)
at org.apache.cassandra.config.CFMetaData.addColumnDefinitionSchema(CFMetaData.java:1261)
at org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1193)
at org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:291)
at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:272)
at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:158)
at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:533)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)

Posted 10 months ago #
xedin
Member

I'm pretty sure that this is related to https://issues.apache.org/jira/browse/CASSANDRA-4356

see https://issues.apache.org/jira/browse/CASSANDRA-4307 (fixed in 1.1.2)

Posted 10 months ago #
ioncube
Member

Thanks for a quick reply Xedin. Just to repeat the question though, how can we get 1.1.0 going again without significant data loss from the current situation?

We just started due diligence with Cassandra on a single server, and that involves not just seeing how it behaves when everything is running fine but also how to deal with system problems. So I'm pleased to see that the server has failed on day one of testing because it gives us a good opportunity to get into that aspect early on.

Would recovery at this point require restoring from a backup, and could that be just of schema related tables, or are there tools to modify the schema/other files without needing the server up? 1.1.0 won't be the last version where some situation makes the server fails to start, so it will be good to know what recovery options there are when it happens.

Posted 10 months ago #
xedin
Member

You can try deleting everything from CASSANDRA_DATA_DIR/system/schema_columnfamilies on all of the nodes and when the node starts back up you can re-create schema.

Posted 10 months ago #
ioncube
Member

Thanks again Xedin, and I did get it going again which is encouraging. I deleted the most recent version of files in system/schema_columns, restarted and it worked, but it failed again at the next restart. I repeated and also removed the latest files in schema_columnfamilies, and it's back as it was.

Posted 10 months ago #
xedin
Member

system.schema_keyspace - holds information about user-defined keyspaces and their attributes, system.schema_columnfamilies - holds information about columns families associated with keyspace and system.schema_columns - holds information about columns per-CF that are present in column_metadata.

For more info on Cassandra's schema take a look at http://www.datastax.com/dev/blog/the-schema-management-renaissance

Posted 10 months ago #

RSS feed for this topic

Reply

You must log in to post.