http://www.datastax.com/support-forums/topic/pig-column-count-limit Software, Support, and Training for Apache Cassandra en-US Fri, 24 May 2013 22:50:09 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/pig-column-count-limit#post-903 Wed, 21 Dec 2011 09:09:16 +0000 Anonymous 903@http://www.datastax.com/support-forums/ <p>Hi everybody,<br /> I have installed DSE 1.0.1 on 3 nodes cluster to test a pig map/reduce job.<br /> I want to count columns for each row of a ColumnFamily with this script :</p> <p>rows = LOAD 'cassandra://&lt;keyspace&gt;/&lt;columnfamily&gt;' USING CassandraStorage();<br /> counted = FOREACH rows GENERATE $0,COUNT_STAR($1);<br /> STORE counted INTO '/tmp/column-count-result' USING PigStorage();</p> <p>It works fine, but there is a default limit of column number loaded (1024)<br /> So I replace my LOAD block by this :</p> <p>rows = LOAD 'cassandra://&lt;keyspace&gt;/&lt;columnfamily&gt;?limit=2147483647' USING CassandraStorage();</p> <p>with 2147483647 = Integer.MAX_VALUE = max column number in a row of a ColumnFamily.</p> <p>Unfortunately, it appears some TimedOutException and OutOfMemoryError: Java heap space in Cassandra logs, and the job doesn't succeed.<br /> So I try to play with the config like indicated here :</p> <p><a href="http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting" rel="nofollow">http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting</a></p> <p>But the result is the same.</p> <p>Someboby have an idea on how configure DSE in order to count a large number of column ?</p> <p>Thanks </p>