I am attempting to upgrade from DSE 2.0-1 to DSE 2.2. I have a development 2-node cluster in VMs running CentOS 6.3.
After the upgrade I am seeing the following in the logs (included below)
I have tried creating a fresh 1-node cluster VM following the DataStax install procedure and still get the same problem. This problem does not appear under DSE 2.0-1.
Note that attempts to use CFS also fail using the dse tool:
[root@dsetest22 DataStax]# dse hadoop fs -ls /
ls: Cannot access /: No such file or directory.
[root@dsetest22 DataStax]# dse hadoop fs -mkdir /test
12/11/02 07:00:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
mkdir: org.apache.thrift.transport.TTransportException
Through trial and error, I've noticed that disabling Solr in the /etc/default/dse file seems to work around this issue (though for development we need both enabled).
Re-enabling Solr afterwards (and restarting dse) results in the system log being spammed again.
From /var/log/cassandra/system.log:
------------
INFO [JOB-TRACKER-INIT] 2012-11-02 06:54:03,579 JobTracker.java (line 2427) problem cleaning system directory: cfs:/tmp/hadoop-cassandra/mapred/system
java.io.IOException: java.lang.IllegalArgumentException
at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.mutateINode(CassandraFileSystemThriftStore.java:918)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.storeINode(CassandraFileSystemThriftStore.java:844)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystem.mkdir(CassandraFileSystem.java:135)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystem.mkdirs(CassandraFileSystem.java:118)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1126)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:336)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2414)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2195)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2189)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:303)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
at org.apache.hadoop.mapred.HadoopTrackerPlugin$1.run(HadoopTrackerPlugin.java:237)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Unknown Source)
at org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:51)
at org.apache.cassandra.db.marshal.DynamicCompositeType.getComparator(DynamicCompositeType.java:88)
at org.apache.cassandra.db.marshal.DynamicCompositeType.getNextComparator(DynamicCompositeType.java:113)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:74)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(Unknown Source)
at java.util.concurrent.ConcurrentSkipListMap.doGet(Unknown Source)
at java.util.concurrent.ConcurrentSkipListMap.get(Unknown Source)
at org.apache.cassandra.db.index.SecondaryIndexManager.getIndexForColumn(SecondaryIndexManager.java:269)
at com.datastax.bdp.cassandra.index.solr.DynamicFieldMutation.getDynamicSolrCore(DynamicFieldMutation.java:48)
at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:783)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.mutateINode(CassandraFileSystemThriftStore.java:914)
... 12 more
