DataStax Support Forums » User Favorites: cko

DataStax Support Forums » User Favorites: cko cko Software, Support, and Training for Apache Cassandra en-US Sun, 19 May 2013 02:08:15 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php cko on "Pig relation schema incorrectly included an extra tuple after creating Cassandra secondary index" http://www.datastax.com/support-forums/topic/pig-relation-schema-incorrectly-included-an-extra-tuple-after-creating-cassandra-secondary-index#post-8316 Mon, 07 Jan 2013 22:10:11 +0000 cko 8316@http://www.datastax.com/support-forums/ I found a solution myself. The extraenous tuple "udid" came from the Cassandra column meta data after a secondary index was created on the column "udid". Looking closer at the output of the "DUMP interaction", Pig did include a tuple for the Cassandra column "udid" but this column was not included in the bag at the end of the relation schema, therefore the FILTER in the FOREACH statement did not include any rows from Cassandra. Removing the FILTER in the FOREACH and referencing the udid.value in the tuple solved the problem. flattenedInteraction = FOREACH interaction { session = FILTER columns by name == 'sessionId'; GENERATE udid.value as udid, flatten(session.value) as sessionId; } schumacr on "Pig unable to connect to Cassandra that requires authentication" http://www.datastax.com/support-forums/topic/pig-unable-to-connect-to-cassandra-that-requires-authentication#post-8314 Mon, 07 Jan 2013 16:48:41 +0000 schumacr 8314@http://www.datastax.com/support-forums/ Typically we allow for a 3-4 month time lag between the time a new C* release occurs and when it is integrated into DSE. This allows the new release to bake in the community plus go through our certification program that ensures it's ready for a production setting. For 1.2, I would look for integration in DSE sometime late Q1 or early Q2. cko on "Pig relation schema incorrectly included an extra tuple after creating Cassandra secondary index" http://www.datastax.com/support-forums/topic/pig-relation-schema-incorrectly-included-an-extra-tuple-after-creating-cassandra-secondary-index#post-8309 Mon, 07 Jan 2013 06:28:53 +0000 cko 8309@http://www.datastax.com/support-forums/ Hi, I have a Cassandra column family InteractionCF. Two of the columns are named udid and sessiondId. I ran a Pig job like this to extract the rows. interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage(); DESCRIBE interaction; DUMP interaction; flattenedInteraction = FOREACH interaction { device = FILTER columns by name == 'udid'; session = FILTER columns by name == 'sessionId'; GENERATE flatten(device.value) as udid, flatten(session.value) as sessionId; } DUMP flattenedInteraction; The Pig relation schema shown by the DESCRIBE function is interaction: {key: chararray,columns: {(name: chararray,value: chararray)}} The output data of the last DUMP was fine. I then ran CLI to create a secondary index on the column udid. update column family InteractionCF with column_metadata = [ {column_name: udid, validation_class: UTF8Type, index_type: KEYS} ]; When the same Pig job was run again, the Pig relation schema has an extra tuple udid interaction: {key: chararray,udid: (name: chararray,value: chararray),columns: {(name: chararray,value: chararray)}} "DUMP interaction" gave the same data as in the first run without secondary index, but "DUMP flattenedInteraction" was empty. Apparently this problem was caused by the extraneous tuple udid in the relation schema which did not match the actual data returned by the Cassandra storage handler. If I tried supplying the schema explicitly in the LOAD statement to exclude the extraneous tuple udid, Pig complaint that the schema is not compatible. grunt> interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage() AS (rowKey:chararray, columns:bag{T:tuple(columnName:chararray, columnValue:chararray)}); 2013-01-07 16:10:52,314 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1031: Incompatable schema: left is "rowKey:chararray,columns:bag{T:tuple(columnName:chararray,columnValue:chararray)}", right is "key:chararray,udid:tuple(name:chararray,value:chararray),columns:bag{:tuple(name:chararray,value:chararray)}" ************ Pig job output without secondary index ******************* grunt> interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage(); grunt> DESCRIBE interaction; interaction: {key: chararray,columns: {(name: chararray,value: chararray)}} grunt> DUMP interaction; 2013-01-07 15:51:25,914 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2013-01-07 15:51:26,268 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2013-01-07 15:51:26,350 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2013-01-07 15:51:26,369 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2013-01-07 15:51:26,369 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2013-01-07 15:51:26,518 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2013-01-07 15:51:26,531 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2013-01-07 15:51:26,532 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job1144508861314187229.jar 2013-01-07 15:51:28,302 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job1144508861314187229.jar created 2013-01-07 15:51:28,317 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2013-01-07 15:51:28,344 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2013-01-07 15:51:28,530 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5 2013-01-07 15:51:28,845 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0020 2013-01-07 15:51:28,845 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: <a href="http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0020" rel="nofollow">http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0020</a> 2013-01-07 15:51:28,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2013-01-07 15:51:39,375 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete 2013-01-07 15:51:40,377 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete 2013-01-07 15:51:42,382 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete 2013-01-07 15:51:43,385 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete 2013-01-07 15:51:45,390 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete 2013-01-07 15:51:53,944 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2013-01-07 15:51:53,945 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: HadoopVersion PigVersion UserId StartedAt FinishedAt Features 1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:51:26 2013-01-07 15:51:53 UNKNOWN Success! Job Stats (time in seconds): JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs job_201212300711_0020 5 0 6 4 5 0 0 0 interaction MAP_ONLY cfs:/tmp/temp-199271653/tmp236958936, Input(s): Successfully read 3 records from: "cassandra://DEVKS/InteractionCF" Output(s): Successfully stored 3 records in: "cfs:/tmp/temp-199271653/tmp236958936" Counters: Total records written : 3 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_201212300711_0020 2013-01-07 15:51:53,953 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! 2013-01-07 15:51:53,974 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5 2013-01-07 15:51:53,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5 (37ce57bd-d09f-4ac3-9b26-2c0bb1fda101,{(sessionId,234),(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0)}) (1111111111-22222222,{(sessionId,222),(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0)}) (20121221-161023.505#996953be-917e-4e8b-9e1c-1a49d54c48d0,{(sessionId,123),(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0)}) grunt> flattenedInteraction = FOREACH interaction { >> device = FILTER columns by name == 'udid'; >> session = FILTER columns by name == 'sessionId'; >> GENERATE >> flatten(device.value) as udid, >> flatten(session.value) as sessionId; >> } grunt> DUMP flattenedInteraction; 2013-01-07 15:51:54,089 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2013-01-07 15:51:54,099 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for interaction: $0 2013-01-07 15:51:54,118 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2013-01-07 15:51:54,120 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2013-01-07 15:51:54,120 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2013-01-07 15:51:54,122 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2013-01-07 15:51:54,123 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2013-01-07 15:51:54,124 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job3897565856870854991.jar 2013-01-07 15:51:55,719 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job3897565856870854991.jar created 2013-01-07 15:51:55,726 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2013-01-07 15:51:55,746 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2013-01-07 15:51:55,830 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5 2013-01-07 15:51:56,246 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0021 2013-01-07 15:51:56,247 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: <a href="http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0021" rel="nofollow">http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0021</a> 2013-01-07 15:51:56,248 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2013-01-07 15:52:06,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete 2013-01-07 15:52:07,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete 2013-01-07 15:52:09,779 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete 2013-01-07 15:52:10,781 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete 2013-01-07 15:52:12,786 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete 2013-01-07 15:52:21,312 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2013-01-07 15:52:21,313 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: HadoopVersion PigVersion UserId StartedAt FinishedAt Features 1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:51:54 2013-01-07 15:52:21 UNKNOWN Success! Job Stats (time in seconds): JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs job_201212300711_0021 5 0 6 4 5 0 0 0 flattenedInteraction,interaction MAP_ONLY cfs:/tmp/temp-199271653/tmp-1710812623, Input(s): Successfully read 3 records from: "cassandra://DEVKS/InteractionCF" Output(s): Successfully stored 3 records in: "cfs:/tmp/temp-199271653/tmp-1710812623" Counters: Total records written : 3 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_201212300711_0021 2013-01-07 15:52:21,319 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! 2013-01-07 15:52:21,326 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5 2013-01-07 15:52:21,326 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5 (996953be-917e-4e8b-9e1c-1a49d54c48d0,123) (996953be-917e-4e8b-9e1c-1a49d54c48d0,234) (996953be-917e-4e8b-9e1c-1a49d54c48d0,222) ************ Pig job output with secondary index ******************* grunt> interaction = LOAD 'cassandra://DEVKS/InteractionCF' USING CassandraStorage(); grunt> DESCRIBE interaction; interaction: {key: chararray,udid: (name: chararray,value: chararray),columns: {(name: chararray,value: chararray)}} grunt> DUMP interaction; 2013-01-07 15:56:15,948 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2013-01-07 15:56:15,981 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2013-01-07 15:56:16,062 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2013-01-07 15:56:16,091 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2013-01-07 15:56:16,092 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2013-01-07 15:56:16,238 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2013-01-07 15:56:16,251 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2013-01-07 15:56:16,251 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2101224938111435909.jar 2013-01-07 15:56:18,003 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2101224938111435909.jar created 2013-01-07 15:56:18,019 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2013-01-07 15:56:18,045 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2013-01-07 15:56:18,441 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5 2013-01-07 15:56:18,545 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2013-01-07 15:56:19,083 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0022 2013-01-07 15:56:19,083 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: <a href="http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0022" rel="nofollow">http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0022</a> 2013-01-07 15:56:29,111 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete 2013-01-07 15:56:30,113 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete 2013-01-07 15:56:32,128 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete 2013-01-07 15:56:33,130 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete 2013-01-07 15:56:35,135 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete 2013-01-07 15:56:43,681 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2013-01-07 15:56:43,683 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: HadoopVersion PigVersion UserId StartedAt FinishedAt Features 1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:56:16 2013-01-07 15:56:43 UNKNOWN Success! Job Stats (time in seconds): JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs job_201212300711_0022 5 0 6 4 5 0 0 0 interaction MAP_ONLY cfs:/tmp/temp-444607507/tmp-1768433647, Input(s): Successfully read 3 records from: "cassandra://DEVKS/InteractionCF" Output(s): Successfully stored 3 records in: "cfs:/tmp/temp-444607507/tmp-1768433647" Counters: Total records written : 3 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_201212300711_0022 2013-01-07 15:56:43,690 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! 2013-01-07 15:56:43,711 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5 2013-01-07 15:56:43,711 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5 (37ce57bd-d09f-4ac3-9b26-2c0bb1fda101,(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0),{(sessionId,234)}) (1111111111-22222222,(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0),{(sessionId,222)}) (20121221-161023.505#996953be-917e-4e8b-9e1c-1a49d54c48d0,(udid,996953be-917e-4e8b-9e1c-1a49d54c48d0),{(sessionId,123)}) grunt> flattenedInteraction = FOREACH interaction { >> device = FILTER columns by name == 'udid'; >> session = FILTER columns by name == 'sessionId'; >> GENERATE >> flatten(device.value) as udid, >> flatten(session.value) as sessionId; >> } grunt> DUMP flattenedInteraction; 2013-01-07 15:56:43,820 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2013-01-07 15:56:43,829 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for interaction: $0, $1 2013-01-07 15:56:43,848 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2013-01-07 15:56:43,850 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2013-01-07 15:56:43,850 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2013-01-07 15:56:43,852 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2013-01-07 15:56:43,853 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2013-01-07 15:56:43,854 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job8930437387737732767.jar 2013-01-07 15:56:45,421 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job8930437387737732767.jar created 2013-01-07 15:56:45,428 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2013-01-07 15:56:45,449 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2013-01-07 15:56:45,526 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 5 2013-01-07 15:56:45,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201212300711_0023 2013-01-07 15:56:45,950 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: <a href="http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0023" rel="nofollow">http://10.69.12.123:50030/jobdetails.jsp?jobid=job_201212300711_0023</a> 2013-01-07 15:56:45,951 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2013-01-07 15:56:55,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete 2013-01-07 15:56:56,976 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete 2013-01-07 15:56:58,980 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 30% complete 2013-01-07 15:56:59,983 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete 2013-01-07 15:57:01,988 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete 2013-01-07 15:57:11,019 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2013-01-07 15:57:11,019 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: HadoopVersion PigVersion UserId StartedAt FinishedAt Features 1.0.2-dse-2.2.1 0.9.2 cassadm 2013-01-07 15:56:43 2013-01-07 15:57:11 UNKNOWN Success! Job Stats (time in seconds): JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs job_201212300711_0023 5 0 6 4 5 0 0 0 flattenedInteraction,interaction MAP_ONLY cfs:/tmp/temp-444607507/tmp-292839261, Input(s): Successfully read 3 records from: "cassandra://DEVKS/InteractionCF" Output(s): Successfully stored 0 records in: "cfs:/tmp/temp-444607507/tmp-292839261" Counters: Total records written : 0 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_201212300711_0023 2013-01-07 15:57:11,025 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! 2013-01-07 15:57:11,036 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 5 2013-01-07 15:57:11,036 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 5 cko on "Pig unable to connect to Cassandra that requires authentication" http://www.datastax.com/support-forums/topic/pig-unable-to-connect-to-cassandra-that-requires-authentication#post-8306 Mon, 07 Jan 2013 00:39:52 +0000 cko 8306@http://www.datastax.com/support-forums/ In <a href="https://issues.apache.org/jira/browse/CASSANDRA-3042" rel="nofollow">https://issues.apache.org/jira/browse/CASSANDRA-3042</a>, authentication support has already been added to the Pig Load funciton CassandraStorage in Cassandra 1.2.0. Currently DSE 2.2.1 includes Cassandra 1.1.6. Is there a schedule when DSE will include Cassandra 1.2.0? Is there any plan to add Pig Cassandra authentication support to DSE when it includes Cassandra 1.2.0? mbulman on "Opscenter - Cluster -List view not showing the correct size" http://www.datastax.com/support-forums/topic/opscenter-cluster-list-view-not-showing-the-correct-size#post-8246 Mon, 31 Dec 2012 16:43:19 +0000 mbulman 8246@http://www.datastax.com/support-forums/ This is most likely caused by a known issue with errors in data collection causing new data from being pushed. Unfortunately, that also causes errors to not be displayed properly, making it harder to diagnose. This should be fixed in the next patch version of OpsCenter (2.1.3), which will be out in the near future. Keep an eye on our dev blog for that announcement. cko on "Opscenter - Cluster -List view not showing the correct size" http://www.datastax.com/support-forums/topic/opscenter-cluster-list-view-not-showing-the-correct-size#post-8223 Sun, 30 Dec 2012 23:29:54 +0000 cko 8223@http://www.datastax.com/support-forums/ Hi, We are also seeing the same symptomp in Ops Center. After restarting the opscenter agent in one node, the disk usage of that one node shown in OpsCenter is correct again. It appears that the agent is not picking up any growth in disk usage since it was started. The agent.log does not have any obvious error messages. There are normal messages like this: INFO [qtp1925049412-39] 2012-12-31 09:08:44,710 HTTP: :get /os-metric/disk-space - 200 We are using DSE 2.1.1, OpsCenter 2.1.2 Chin srk_user on "Opscenter - Cluster -List view not showing the correct size" http://www.datastax.com/support-forums/topic/opscenter-cluster-list-view-not-showing-the-correct-size#post-8207 Sat, 29 Dec 2012 13:36:31 +0000 srk_user 8207@http://www.datastax.com/support-forums/ In OpsCenter - Cluster -List view - the data size is not showing up correctly. The OS Disk Usage for the Cassandra Data File system shows 901GB but on the List view it shows up as only 548GB. Why is the Opscenter List View not showing the correct file size? Sven on "Pig unable to connect to Cassandra that requires authentication" http://www.datastax.com/support-forums/topic/pig-unable-to-connect-to-cassandra-that-requires-authentication#post-8167 Thu, 27 Dec 2012 21:57:51 +0000 Sven 8167@http://www.datastax.com/support-forums/ SimpleAuthenticator and SimpleAuthority are not currently supported by the DSE components. We are investigating the possibility of adding support for enhanced security in a future release of DSE. cko on "ETL Tools to transfer data from Cassandra into other relational databases" http://www.datastax.com/support-forums/topic/etl-tools-to-transfer-data-from-cassandra-into-other-relational-databases#post-8153 Thu, 27 Dec 2012 06:46:11 +0000 cko 8153@http://www.datastax.com/support-forums/ Hi Srini, Thanks for your response. A few months ago, we did try Pentaho briefly. The job caused an OutOfMemoryError in Cassandra. The Cassandra Pentaho plugin does not handle retrieving rows in batches. It was mentioned that the next version of the plugin will address that issue. We will look at it again when it is available. Thanks again. Chin cko on "Pig unable to connect to Cassandra that requires authentication" http://www.datastax.com/support-forums/topic/pig-unable-to-connect-to-cassandra-that-requires-authentication#post-8152 Thu, 27 Dec 2012 06:35:26 +0000 cko 8152@http://www.datastax.com/support-forums/ Hi, I am trying Cassandra authentication using the sample SimpleAuthenticator and SimpleAuthority classes and also trying securing the Cassandra JMX port. Ops Center can successfully connect to and monitor a Cassandra cluster with the authentication enabled. However there are problems running Pig against a Cassandra cluster with authentication enabled. Looking at the Pig stack trace, we found that the SimpleAuthenticator was throwing an exception during initialisation, so we added the required JVM -D system properties to both scripts $DSE_HOME/bin/dsetool and $DSE_HOME/resources/pig/bin/pig. JVM_OPTS="$JVM_OPTS -Dpasswd.properties=$CASSANDRA_HOME/conf/passwd.properties" JVM_OPTS="$JVM_OPTS -Daccess.properties=$CASSANDRA_HOME/conf/access.properties" JVM_OPTS="$JVM_OPTS -Dpasswd.mode=MD5" That takes us further, but then it appears that the CFS custom storage handler received an exception from Cassandra when it tries to connect to the Cassandra database without supplying credentials. Error before Pig is launched ---------------------------- ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:206) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:119) at org.apache.pig.impl.PigContext.connect(PigContext.java:185) at org.apache.pig.PigServer.<init>(PigServer.java:244) at org.apache.pig.PigServer.<init>(PigServer.java:229) at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47) at org.apache.pig.Main.run(Main.java:492) at org.apache.pig.Main.main(Main.java:111) Caused by: java.io.IOException: InvalidRequestException(why:You have not logged in) at com.datastax.bdp.util.CassandraProxyClient.initialize(CassandraProxyClient.java:227) at com.datastax.bdp.util.CassandraProxyClient.<init>(CassandraProxyClient.java:180) at com.datastax.bdp.util.CassandraProxyClient.newProxyConnection(CassandraProxyClient.java:119) at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.initialize(CassandraFileSystemThriftStore.java:195) at com.datastax.bdp.hadoop.cfs.CassandraFileSystem.initialize(CassandraFileSystem.java:67) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72) ... 9 more Caused by: InvalidRequestException(why:You have not logged in) at org.apache.cassandra.thrift.Cassandra$describe_keyspaces_result.read(Cassandra.java:22379) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_keyspaces(Cassandra.java:1004) at org.apache.cassandra.thrift.Cassandra$Client.describe_keyspaces(Cassandra.java:992) at com.datastax.bdp.util.CassandraProxyClient.initialize(CassandraProxyClient.java:213) ... 19 more I could not find anything in Datastax documentation on how to supply credentials in Pig to connect to Cassandra. Does the current Cassandra storage handler support connecting to a secured Cassandra database? We are using DSE 2.2.1. Thanks in advance. Chin Srini on "ETL Tools to transfer data from Cassandra into other relational databases" http://www.datastax.com/support-forums/topic/etl-tools-to-transfer-data-from-cassandra-into-other-relational-databases#post-8015 Tue, 18 Dec 2012 18:09:37 +0000 Srini 8015@http://www.datastax.com/support-forums/ This should help you. It's just an overview. Drill down more as per your requirement. <a href="http://ovum.com/2012/03/08/pentaho-expands-big-data-coverage/" rel="nofollow">http://ovum.com/2012/03/08/pentaho-expands-big-data-coverage/</a> <a href="http://wiki.pentaho.com/display/DATAMINING/Cassandra+Source+and+Sink+in+Weka" rel="nofollow">http://wiki.pentaho.com/display/DATAMINING/Cassandra+Source+and+Sink+in+Weka</a> cko on "ETL Tools to transfer data from Cassandra into other relational databases" http://www.datastax.com/support-forums/topic/etl-tools-to-transfer-data-from-cassandra-into-other-relational-databases#post-7972 Fri, 14 Dec 2012 03:13:03 +0000 cko 7972@http://www.datastax.com/support-forums/ We will use Cassandra as logging storage in one of our web application. The application only insert rows into Cassandra but never update or delete any rows. The CF is expected to grow by about 0.5 million rows per day. We need to transfer the data in Cassandra to another relational database daily. Due to the large size of the CF, instead of truncating the relational table and reloading all rows into it each time, we plan to run a job to select the "delta" rows since the last run and insert them into the relational database. We know we can use Java, Pig or Hive to extract the delta rows to a flat file and load the data into the target relational table. We are particularly interested in a process that can extract delta rows without scanning the entire CF. Has anyone used any other ETL tools to do this kind of delta extraction from Cassandra? We appreciate any comments and experience. Thanks, Chin nickmbailey on "Selecting rows efficiently from a Cassandra CF containing time series data" http://www.datastax.com/support-forums/topic/selecting-rows-efficiently-from-a-cassandra-cf-containing-time-series-data#post-7947 Tue, 11 Dec 2012 16:30:53 +0000 nickmbailey 7947@http://www.datastax.com/support-forums/ I would recommend reading: <a href="http://www.datastax.com/dev/blog/advanced-time-series-with-cassandra" rel="nofollow">http://www.datastax.com/dev/blog/advanced-time-series-with-cassandra</a> You likely will want to restructure your data model so that events are stored in a row ordered by time. You could then partition this row by some time value to get better distribution. nickmbailey on "Selecting rows efficiently from a Cassandra CF containing time series data" http://www.datastax.com/support-forums/topic/selecting-rows-efficiently-from-a-cassandra-cf-containing-time-series-data#post-7946 Tue, 11 Dec 2012 16:30:52 +0000 nickmbailey 7946@http://www.datastax.com/support-forums/ I would recommend reading: <a href="http://www.datastax.com/dev/blog/advanced-time-series-with-cassandra" rel="nofollow">http://www.datastax.com/dev/blog/advanced-time-series-with-cassandra</a> You likely will want to restructure your data model so that events are stored in a row ordered by time. You could then partition this row by some time value to get better distribution. cko on "Selecting rows efficiently from a Cassandra CF containing time series data" http://www.datastax.com/support-forums/topic/selecting-rows-efficiently-from-a-cassandra-cf-containing-time-series-data#post-7940 Tue, 11 Dec 2012 13:50:51 +0000 cko 7940@http://www.datastax.com/support-forums/ I would like to get some opinions on how to select an incremental range of rows efficiently from a Cassandra CF containing time series data. Background: We have a web application that uses a Cassandra CF as logging storage. We insert a row into the CF for every "event" of each user of the web application. The row key is timestamp+userid. The column values are unstructured data. We only insert rows but never update or delete any rows in the CF. Data volume: The CF grows by about 0.5 million rows per day. We have a 4 node cluster and use the RandomPartitioner to spread the rows across the nodes. Requirements: There is a need to transfer the Cassandra data to another relational database periodically. Due to the large size of the CF, instead of truncating the relational table and reloading all rows into it each time, we plan to run a job to select the "delta" rows since the last run and insert them into the relational database. We would like to have some flexibility in how often the data transfer job is done. It may be run several times each day, or it may be not run at all on a day. Options considered: - We are using RandomPartitioner, so range scan by row key is not feasible. - Add a secondary index on the timestamp column, but reading rows via secondary index still requires an equality condition and does not support range scan. - Add a secondary index on a column containing the date and hour of the timestamp. Iterate each hour between the time job was last run and now. Fetch all rows of each hour. I would appreciate any ideas of other design options of the Cassandra CF to enable extracting the rows efficiently. Besides Java, has anyone used any ETL tools to do this kind of delta extraction from Cassandra?