http://www.datastax.com/support-forums/topic/agregate-data-by-columns-in-pig Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 01:59:16 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/agregate-data-by-columns-in-pig#post-1828 Thu, 03 May 2012 15:28:19 +0000 Anonymous 1828@http://www.datastax.com/support-forums/ <p>Would it help to flatten the bag out first? then you can use of foreach or group to re-arrange. </p> <p>e.g.<br /> <code><br /> p1 = LOAD 'cassandra://keyspace/CF' USING CassandraStorage() AS (key:chararray, columns: bag{T: tuple(property:chararray, value:chararray)});<br /> p2 = FOREACH p1 GENERATE key, flatten (columns);<br /> </code></p> <p>You'll get a tuples of (key, property, value) and can then filter, split, group by, etc..</p> <p>Although, let me know if you're able use GROUP BY or JOIN on the key, I keep getting an error - recently created a post, but no replies yet. </p> http://www.datastax.com/support-forums/topic/agregate-data-by-columns-in-pig#post-1821 Wed, 02 May 2012 16:47:39 +0000 Anonymous 1821@http://www.datastax.com/support-forums/ <p>Probably the only way would be to add all/some of the columns to the metadata so they become separated in the Pig schema. </p> http://www.datastax.com/support-forums/topic/agregate-data-by-columns-in-pig#post-1820 Wed, 02 May 2012 16:35:12 +0000 Anonymous 1820@http://www.datastax.com/support-forums/ <p>I have data in Cassandra in following structure:</p> <pre><code>[default@data] list values; Using default limit of 100 ------------------- RowKey: 123:1333065600000 =&gt; (column=16000, value=331, timestamp=1333724446041000) =&gt; (column=76000, value=314, timestamp=1333724446042000) =&gt; (column=136000, value=333, timestamp=1333724446043000) RowKey: 123:1332979200000 =&gt; (column=6000, value=300, timestamp=1333743661692000) =&gt; (column=66000, value=302, timestamp=1333743661692001) =&gt; (column=126000, value=303, timestamp=1333743661694000) =&gt; (column=186000, value=234, timestamp=1333743661695000) =&gt; (column=246000, value=445, timestamp=1333743661696000) =&gt; (column=306000, value=331, timestamp=1333743661696001) =&gt; (column=366000, value=455, timestamp=1333743661698000) RowKey: 121:1334102400000 =&gt; (column=68608000, value=12, timestamp=1334173122715000) =&gt; (column=68668000, value=12, timestamp=1334173122715001)</code></pre> <p>In other words - number of columns can differ row per row. Now I need to do aggregation using pig of all values. I cannot change the structure. Any idea how to do that? </p>