http://www.datastax.com/support-forums/topic/cassandra-crashes-when-running-a-import-script-for-big-data Software, Support, and Training for Apache Cassandra en-US Mon, 20 May 2013 13:44:56 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/cassandra-crashes-when-running-a-import-script-for-big-data#post-1754 Tue, 24 Apr 2012 15:40:33 +0000 Anonymous 1754@http://www.datastax.com/support-forums/ <p>Dear All,</p> <p>I'm working on an import script that process and import very big data (over 3GB data with million records) into a Cassandra keyspace. The script always crashes after importing around 2,000,000 records with errors below</p> <p>^[[0K - test_application: 1727062<br /> ^[[0K - test_application: 1727063/usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/socket.rb:109:in `read': CassandraThrift::Cassandra::Client::TransportException<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/base_transport.rb:87:in `read_all'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/framed_transport.rb:104:in `read_frame'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/framed_transport.rb:69:in `read_into_buffer'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/protocol/binary_protocol.rb:194:in `read_i32'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/protocol/binary_protocol.rb:120:in `read_message_begin'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/client.rb:45:in `receive_message'<br /> from /home/deployer/.bundler/ruby/1.9.1/cassandra-4bca4c8f0dc7/vendor/1.0/gen-rb/cassandra.rb:111:in `recv_multiget_slice'<br /> from /home/deployer/.bundler/ruby/1.9.1/cassandra-4bca4c8f0dc7/vendor/1.0/gen-rb/cassandra.rb:103:in `multiget_slice'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift_client-0.8.1/lib/thrift_client/abstract_thrift_client.rb:150:in `handled_proxy'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/thrift_client-0.8.1/lib/thrift_client/abstract_thrift_client.rb:60:in `multiget_slice'<br /> from /home/deployer/.bundler/ruby/1.9.1/cassandra-4bca4c8f0dc7/lib/cassandra/protocol.rb:83:in `_multiget'<br /> from /home/deployer/.bundler/ruby/1.9.1/cassandra-4bca4c8f0dc7/lib/cassandra/cassandra.rb:619:in `multi_get'<br /> from /usr/local/lib/ruby/gems/1.9.1/gems/active_column-0.2/lib/active_column/base.rb:25:in `find'</p> <p>I've already tried some ways below I found from Googling but no luck, please help</p> <pre><code>*. Fix error (this not work) thrift/transport/socket.rb:109:in</code></pre> read': CassandraThrift::Cassandra::Client::TransportException</p> <p> That's a Ruby 1.9 issue.<br /> As suggested we convert non ASCII strings to binary before writing them: In ~/.rvm/gems/ruby-1.9.2-p290/gems/thrift-0.7.0/lib/thrift/protocol/binary_protocol.rb, this is a patch suggestion:</p> <p> sudo nano /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/protocol/binary_protocol.rb</p> <p> {code}<br /> def write_string(str)<br /> if str.encoding.to_s != "US-ASCII"<br /> str = str.unpack("a*").first<br /> end<br /> write_i32(str.length)<br /> trans.write(str)<br /> end<br /> {code} </p> <p>*. Concurrent requests crash operations(seem not work)<br /> /Users/mseeger/.rvm/gems/ruby-1.9.3-rc1/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:109:in `read': CassandraThrift::Cassandra::Client::TransportException</p> <p> I have two ruby processes: one process bulk inserting data (long running), the other process searching (manually launched during the insert)</p> <p> If I launch a search during the insertion, one of the two jobs will die. </p> <p> I think it MIGHT have something to do with the way I inserted the data.<br /> When I was doing a batch insertion (500 at a time) it did crash. When I am inserting them 1 by 1 (slower), these crashes don't seem to happen. The search operation stayed the same.</p> <p> Ok, running the client with {:retries =&gt; 10, :timeout =&gt; 15, :connect_timeout =&gt; 15} seems to solve the problem. I guess it was just a question of connection defaults. For a single node development "cluster" they seem to be too low.</p> <p>*. Update /etc/cassandra/conf/cassandra.yml config<br /> # Frame size for thrift (maximum field length).<br /> # 0 disables TFramedTransport in favor of TSocket. This option<br /> # is deprecated; we strongly recommend using Framed mode.<br /> thrift_framed_transport_size_in_mb: 150</p> <p> # The max length of a thrift message, including all fields and<br /> # internal thrift overhead.<br /> thrift_max_message_length_in_mb: 160<br /> ` </p>