Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

Pig relation schema incorrectly included an extra tuple after creating Cassandra secondary index

(2 posts) (1 voice)

Started 4 months ago by cko
Latest reply from cko
This topic is not a support question

Tags:

No tags yet.

cko
Member

Hi,

I have a Cassandra column family InteractionCF. Two of the columns are named udid and sessiondId.

I ran a Pig job like this to extract the rows.

interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage();
DESCRIBE interaction;
DUMP interaction;
flattenedInteraction = FOREACH interaction {
device = FILTER columns by name == 'udid';
session = FILTER columns by name == 'sessionId';
GENERATE
flatten(device.value) as udid,
flatten(session.value) as sessionId;
}
DUMP flattenedInteraction;

The Pig relation schema shown by the DESCRIBE function is
interaction: {key: chararray,columns: {(name: chararray,value: chararray)}}

The output data of the last DUMP was fine.

I then ran CLI to create a secondary index on the column udid.

update column family InteractionCF with
column_metadata =
[
{column_name: udid, validation_class: UTF8Type, index_type: KEYS}
];

When the same Pig job was run again, the Pig relation schema has an extra tuple udid

interaction: {key: chararray,udid: (name: chararray,value: chararray),columns: {(name: chararray,value: chararray)}}

"DUMP interaction" gave the same data as in the first run without secondary index, but "DUMP flattenedInteraction" was empty. Apparently this problem was caused by the extraneous tuple udid in the relation schema which did not match the actual data returned by the Cassandra storage handler.

If I tried supplying the schema explicitly in the LOAD statement to exclude the extraneous tuple udid, Pig complaint that the schema is not compatible.

grunt> interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage() AS (rowKey:chararray, columns:bag{T:tuple(columnName:chararray, columnValue:chararray)});

2013-01-07 16:10:52,314 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1031: Incompatable schema: left is "rowKey:chararray,columns:bag{T:tuple(columnName:chararray,columnValue:chararray)}", right is "key:chararray,udid:tuple(name:chararray,value:chararray),columns:bag{:tuple(name:chararray,value:chararray)}"

************ Pig job output without secondary index *******************

grunt> interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage();
grunt> DESCRIBE interaction;
interaction: {key: chararray,columns: {(name: chararray,value: chararray)}}
grunt> DUMP interaction;
2013-01-07 15:51:25,914 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-01-07 15:51:26,268 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-01-07 15:51:26,350 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-01-07 15:51:26,369 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-01-07 15:51:26,369 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-01-07 15:51:26,518 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-01-07 15:51:26,531 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-01-07 15:51:26,532 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job1144508861314187229.jar
2013-01-07 15:51:28,302 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job1144508861314187229.jar created
2013-01-07 15:51:28,317 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-01-07 15:51:28,344 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-01-07 15:51:28,530 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5
2013-01-07 15:51:28,845 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0020
2013-01-07 15:51:28,845 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0020
2013-01-07 15:51:28,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-01-07 15:51:39,375 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete
2013-01-07 15:51:40,377 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete
2013-01-07 15:51:42,382 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete
2013-01-07 15:51:43,385 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete
2013-01-07 15:51:45,390 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2013-01-07 15:51:53,944 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-01-07 15:51:53,945 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:51:26 2013-01-07 15:51:53 UNKNOWN

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201212300711_0020 5 0 6 4 5 0 0 0 interaction MAP_ONLY cfs:/tmp/temp-199271653/tmp236958936,

Input(s):
Successfully read 3 records from: "cassandra://DEVKS/InteractionCF"

Output(s):
Successfully stored 3 records in: "cfs:/tmp/temp-199271653/tmp236958936"

Counters:
Total records written : 3
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201212300711_0020

2013-01-07 15:51:53,953 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2013-01-07 15:51:53,974 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5
2013-01-07 15:51:53,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5

(37ce57bd-d09f-4ac3-9b26-2c0bb1fda101,{(sessionId,234),(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0)})
(1111111111-22222222,{(sessionId,222),(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0)})
(20121221-161023.505#996953be-917e-4e8b-9e1c-1a49d54c48d0,{(sessionId,123),(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0)})

grunt> flattenedInteraction = FOREACH interaction {
>> device = FILTER columns by name == 'udid';
>> session = FILTER columns by name == 'sessionId';
>> GENERATE
>> flatten(device.value) as udid,
>> flatten(session.value) as sessionId;
>> }
grunt> DUMP flattenedInteraction;
2013-01-07 15:51:54,089 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-01-07 15:51:54,099 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for interaction: $0
2013-01-07 15:51:54,118 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-01-07 15:51:54,120 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-01-07 15:51:54,120 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-01-07 15:51:54,122 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-01-07 15:51:54,123 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-01-07 15:51:54,124 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job3897565856870854991.jar
2013-01-07 15:51:55,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job3897565856870854991.jar created
2013-01-07 15:51:55,726 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-01-07 15:51:55,746 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-01-07 15:51:55,830 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5
2013-01-07 15:51:56,246 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0021
2013-01-07 15:51:56,247 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0021
2013-01-07 15:51:56,248 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-01-07 15:52:06,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete
2013-01-07 15:52:07,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete
2013-01-07 15:52:09,779 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete
2013-01-07 15:52:10,781 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete
2013-01-07 15:52:12,786 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2013-01-07 15:52:21,312 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-01-07 15:52:21,313 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:51:54 2013-01-07 15:52:21 UNKNOWN

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201212300711_0021 5 0 6 4 5 0 0 0 flattenedInteraction,interaction MAP_ONLY cfs:/tmp/temp-199271653/tmp-1710812623,

Input(s):
Successfully read 3 records from: "cassandra://DEVKS/InteractionCF"

Output(s):
Successfully stored 3 records in: "cfs:/tmp/temp-199271653/tmp-1710812623"

Counters:
Total records written : 3
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201212300711_0021

2013-01-07 15:52:21,319 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2013-01-07 15:52:21,326 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5
2013-01-07 15:52:21,326 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5

(996953be-917e-4e8b-9e1c-1a49d54c48d0,123)
(996953be-917e-4e8b-9e1c-1a49d54c48d0,234)
(996953be-917e-4e8b-9e1c-1a49d54c48d0,222)

************ Pig job output with secondary index *******************

grunt> interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage();

grunt> DESCRIBE interaction;
interaction: {key: chararray,udid: (name: chararray,value: chararray),columns: {(name: chararray,value: chararray)}}

grunt> DUMP interaction;
2013-01-07 15:56:15,948 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-01-07 15:56:15,981 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-01-07 15:56:16,062 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-01-07 15:56:16,091 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-01-07 15:56:16,092 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-01-07 15:56:16,238 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-01-07 15:56:16,251 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-01-07 15:56:16,251 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2101224938111435909.jar
2013-01-07 15:56:18,003 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2101224938111435909.jar created
2013-01-07 15:56:18,019 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-01-07 15:56:18,045 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-01-07 15:56:18,441 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5
2013-01-07 15:56:18,545 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-01-07 15:56:19,083 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0022
2013-01-07 15:56:19,083 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0022
2013-01-07 15:56:29,111 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete
2013-01-07 15:56:30,113 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete
2013-01-07 15:56:32,128 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete
2013-01-07 15:56:33,130 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete
2013-01-07 15:56:35,135 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2013-01-07 15:56:43,681 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-01-07 15:56:43,683 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:56:16 2013-01-07 15:56:43 UNKNOWN

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201212300711_0022 5 0 6 4 5 0 0 0 interaction MAP_ONLY cfs:/tmp/temp-444607507/tmp-1768433647,

Input(s):
Successfully read 3 records from: "cassandra://DEVKS/InteractionCF"

Output(s):
Successfully stored 3 records in: "cfs:/tmp/temp-444607507/tmp-1768433647"

Counters:
Total records written : 3
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201212300711_0022

2013-01-07 15:56:43,690 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2013-01-07 15:56:43,711 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5
2013-01-07 15:56:43,711 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5

(37ce57bd-d09f-4ac3-9b26-2c0bb1fda101,(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0),{(sessionId,234)})
(1111111111-22222222,(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0),{(sessionId,222)})
(20121221-161023.505#996953be-917e-4e8b-9e1c-1a49d54c48d0,(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0),{(sessionId,123)})

grunt> flattenedInteraction = FOREACH interaction {
>> device = FILTER columns by name == 'udid';
>> session = FILTER columns by name == 'sessionId';
>> GENERATE
>> flatten(device.value) as udid,
>> flatten(session.value) as sessionId;
>> }
grunt> DUMP flattenedInteraction;
2013-01-07 15:56:43,820 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-01-07 15:56:43,829 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for interaction: $0, $1
2013-01-07 15:56:43,848 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-01-07 15:56:43,850 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-01-07 15:56:43,850 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-01-07 15:56:43,852 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-01-07 15:56:43,853 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-01-07 15:56:43,854 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job8930437387737732767.jar
2013-01-07 15:56:45,421 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job8930437387737732767.jar created
2013-01-07 15:56:45,428 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-01-07 15:56:45,449 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-01-07 15:56:45,526 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5
2013-01-07 15:56:45,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0023
2013-01-07 15:56:45,950 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0023
2013-01-07 15:56:45,951 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-01-07 15:56:55,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete
2013-01-07 15:56:56,976 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete
2013-01-07 15:56:58,980 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete
2013-01-07 15:56:59,983 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete
2013-01-07 15:57:01,988 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2013-01-07 15:57:11,019 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-01-07 15:57:11,019 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:56:43 2013-01-07 15:57:11 UNKNOWN

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201212300711_0023 5 0 6 4 5 0 0 0 flattenedInteraction,interaction MAP_ONLY cfs:/tmp/temp-444607507/tmp-292839261,

Input(s):
Successfully read 3 records from: "cassandra://DEVKS/InteractionCF"

Output(s):
Successfully stored 0 records in: "cfs:/tmp/temp-444607507/tmp-292839261"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201212300711_0023

2013-01-07 15:57:11,025 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2013-01-07 15:57:11,036 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5
2013-01-07 15:57:11,036 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5

Posted 4 months ago #
cko
Member

I found a solution myself.

The extraenous tuple "udid" came from the Cassandra column meta data after a secondary index was created on the column "udid".

Looking closer at the output of the "DUMP interaction", Pig did include a tuple for the Cassandra column "udid" but this column was not included in the bag at the end of the relation schema, therefore the FILTER in the FOREACH statement did not include any rows from Cassandra.

Removing the FILTER in the FOREACH and referencing the udid.value in the tuple solved the problem.

flattenedInteraction = FOREACH interaction {
session = FILTER columns by name == 'sessionId';
GENERATE
udid.value as udid,
flatten(session.value) as sessionId;
}

Posted 4 months ago #

RSS feed for this topic

Reply

You must log in to post.