DataStax Support Forums » Topic: Brisk pig

DataStax Support Forums » Topic: Brisk pig - cfs: seems to be failing http://www.datastax.com/support-forums/topic/brisk-pig-cfs-seems-to-be-failing Software, Support, and Training for Apache Cassandra en-US Wed, 22 May 2013 11:05:58 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php slatemine on "Brisk pig - cfs: seems to be failing" http://www.datastax.com/support-forums/topic/brisk-pig-cfs-seems-to-be-failing#post-290 Fri, 08 Jul 2011 08:01:28 +0000 slatemine 290@http://www.datastax.com/support-forums/ I thinks I've found at least a partial solution. The second error Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=seeds for was caused by (class)path precedence - I happened to have an old cassandra.yaml in my $HOME directory. brisk hadoop fs -ls was finding that one first and failing. I haven't verified this but I think grunt may have done the same for data output p slatemine on "Brisk pig - cfs: seems to be failing" http://www.datastax.com/support-forums/topic/brisk-pig-cfs-seems-to-be-failing#post-253 Fri, 01 Jul 2011 15:08:11 +0000 slatemine 253@http://www.datastax.com/support-forums/ I am running single node - but with a listen address != localhost I think that this is stopping cfs:// from connecting correctly for IO operations. Two examples below, one from pig, the second from command line brisk hadoop fs -ls The second one looks to be a parsing error. p --- LOG in PIG grunt> entities = LOAD 'cassandra://CassBuilderPRJT/rtbEnts' USING CassandraStorage() AS (ents,columns: bag {T: tuple(col, val)}); grunt> B = GROUP entities by ents; grunt> X = FOREACH B GENERATE COUNT(entities); grunt> dump X; 2011-07-01 15:22:34,242 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY 2011-07-01 15:22:34,243 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used. 2011-07-01 15:22:34,514 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: X: Store(cfs://null/tmp/temp1522760494/tmp1523872576:org.apache.pig.impl.io.InterStorage) - scope-15 Operator Key: scope-15) 2011-07-01 15:22:34,536 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2011-07-01 15:22:34,552 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner 2011-07-01 15:22:34,606 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2011-07-01 15:22:34,606 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2011-07-01 15:22:34,845 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2011-07-01 15:22:34,875 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2011-07-01 15:22:36,827 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2011-07-01 15:22:36,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=0 2011-07-01 15:22:36,865 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1 2011-07-01 15:22:36,926 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2011-07-01 15:22:37,427 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2011-07-01 15:22:38,081 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 181 2011-07-01 15:22:38,814 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201107011409_0006 2011-07-01 15:22:38,814 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: <a href="http://10.20.1.205:50030/jobdetails.jsp?jobid=job_201107011409_0006" rel="nofollow">http://10.20.1.205:50030/jobdetails.jsp?jobid=job_201107011409_0006</a> 2011-07-01 15:24:35,461 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 4% complete 2011-07-01 15:29:24,091 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 15% complete 2011-07-01 15:40:09,489 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 40% complete 2011-07-01 15:49:44,907 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 63% complete 2011-07-01 15:51:08,813 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 83% complete 2011-07-01 15:52:24,638 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2011-07-01 15:52:24,640 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics: HadoopVersion PigVersion UserId StartedAt FinishedAt Features 0.20.203.1-brisk1-beta2 0.8.3 prjt 2011-07-01 15:22:34 2011-07-01 15:52:24 GROUP_BY Success! Job Stats (time in seconds): JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs job_201107011409_0006 181 1 27 12 18 1677 1677 1677 B,X,entities GROUP_BY,COMBINER cfs://null/tmp/temp1522760494/tmp1523872576, Input(s): Successfully read 11900289 records from: "cassandra://CassBuilderPRJT/rtbEnts" Output(s): Successfully stored 11900289 records in: "cfs://null/tmp/temp1522760494/tmp1523872576" Counters: Total records written : 11900289 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_201107011409_0006 2011-07-01 15:52:24,707 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! 2011-07-01 15:52:24,756 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2011-07-01 15:52:24,756 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 2011-07-01 15:52:24,816 [main] ERROR org.apache.pig.backend.hadoop.executionengine.HJob - java.lang.RuntimeException: Local file does not exist: /var/lib/cassandra/data/cfs/sblocks-g-41-Data.db 2011-07-01 15:52:24,817 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. java.lang.RuntimeException: Local file does not exist: /var/lib/cassandra/data/cfs/sblocks-g-41-Data.db 2011-07-01 15:52:24,817 [main] WARN org.apache.pig.tools.grunt.Grunt - There is no log file to write to. 2011-07-01 15:52:24,817 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.Error: java.lang.RuntimeException: Local file does not exist: /var/lib/cassandra/data/cfs/sblocks-g-41-Data.db at org.apache.pig.backend.hadoop.executionengine.HJob$1.hasNext(HJob.java:118) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:616) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76) at org.apache.pig.Main.run(Main.java:455) at org.apache.pig.Main.main(Main.java:107) Caused by: java.lang.RuntimeException: Local file does not exist: /var/lib/cassandra/data/cfs/sblocks-g-41-Data.db at org.apache.cassandra.hadoop.fs.CassandraFileSystemThriftStore.readLocalBlock(CassandraFileSystemThriftStore.java:477) at org.apache.cassandra.hadoop.fs.CassandraFileSystemThriftStore.retrieveSubBlock(CassandraFileSystemThriftStore.java:398) at org.apache.cassandra.hadoop.fs.CassandraSubBlockInputStream.subBlockSeekTo(CassandraSubBlockInputStream.java:140) at org.apache.cassandra.hadoop.fs.CassandraSubBlockInputStream.read(CassandraSubBlockInputStream.java:90) at org.apache.cassandra.hadoop.fs.CassandraInputStream.read(CassandraInputStream.java:135) at java.io.DataInputStream.read(DataInputStream.java:132) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at org.apache.pig.impl.io.BufferedPositionedInputStream.read(BufferedPositionedInputStream.java:52) at org.apache.pig.impl.io.InterRecordReader.nextKeyValue(InterRecordReader.java:86) at org.apache.pig.impl.io.InterStorage.getNext(InterStorage.java:77) at org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:209) at org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:189) at org.apache.pig.backend.hadoop.executionengine.HJob$1.hasNext(HJob.java:111) ... 7 more --- missing seeds ? > brisk hadoop fs -ls 11/07/01 16:03:49 ERROR config.DatabaseDescriptor: Fatal configuration error error Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=seeds for JavaBean=org.apache.cassandra.config.Config@5c1428ea; Unable to find property 'seeds' on class: org.apache.cassandra.config.Config in "<reader>", line 10, column 1: cluster_name: 'Test Cluster' ^ at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:136) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122) at org.yaml.snakeyaml.Loader.load(Loader.java:52) at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:138) at org.apache.cassandra.utils.FBUtilities.getLocalAddress(FBUtilities.java:121) at org.apache.cassandra.hadoop.fs.CassandraFileSystemThriftStore.initialize(CassandraFileSystemThriftStore.java:128) at org.apache.cassandra.hadoop.fs.CassandraFileSystem.initialize(CassandraFileSystem.java:59) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1310) at org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:65) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1328) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109) at org.apache.hadoop.fs.FsShell.init(FsShell.java:82) at org.apache.hadoop.fs.FsShell.run(FsShell.java:1745) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895) Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=seeds for JavaBean=org.apache.cassandra.config.Config@5c1428ea; Unable to find property 'seeds' on class: org.apache.cassandra.config.Config at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184) at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370) ... 19 more Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'seeds' on class: org.apache.cassandra.config.Config at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.getProperty(Constructor.java:342) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:240) ... 21 more null; Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=seeds for JavaBean=org.apache.cassandra.config.Config@5c1428ea; Unable to find property 'seeds' on class: org.apache.cassandra.config.Config Invalid yaml; unable to start server. See log for stacktrace.