Aggregate data by columns in Pig

(3 posts) (3 voices)

• Started 1 year ago by
• Latest reply from
• This topic is not resolved

Tags:

• pig

1. 

   Anonymous
   Unregistered

   I have data in Cassandra in following structure:

   [default@data] list values;
   Using default limit of 100
   -------------------
   RowKey: 123:1333065600000
   => (column=16000, value=331, timestamp=1333724446041000)
   => (column=76000, value=314, timestamp=1333724446042000)
   => (column=136000, value=333, timestamp=1333724446043000)
   RowKey: 123:1332979200000
   => (column=6000, value=300, timestamp=1333743661692000)
   => (column=66000, value=302, timestamp=1333743661692001)
   => (column=126000, value=303, timestamp=1333743661694000)
   => (column=186000, value=234, timestamp=1333743661695000)
   => (column=246000, value=445, timestamp=1333743661696000)
   => (column=306000, value=331, timestamp=1333743661696001)
   => (column=366000, value=455, timestamp=1333743661698000)
   RowKey: 121:1334102400000
   => (column=68608000, value=12, timestamp=1334173122715000)
   => (column=68668000, value=12, timestamp=1334173122715001)

   In other words - number of columns can differ row per row. Now I need to do aggregation using pig of all values. I cannot change the structure. Any idea how to do that?

   Posted 1 year ago #
2. 

   Anonymous
   Unregistered

   Probably the only way would be to add all/some of the columns to the metadata so they become separated in the Pig schema.

   Posted 1 year ago #
3. 

   Anonymous
   Unregistered

   Would it help to flatten the bag out first? then you can use of foreach or group to re-arrange.

   e.g.
   
p1 = LOAD 'cassandra://keyspace/CF' USING CassandraStorage() AS (key:chararray, columns: bag{T: tuple(property:chararray, value:chararray)});
p2 = FOREACH p1 GENERATE key, flatten (columns);


   You'll get a tuples of (key, property, value) and can then filter, split, group by, etc..

   Although, let me know if you're able use GROUP BY or JOIN on the key, I keep getting an error - recently created a post, but no replies yet.

   Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.