DataStax Support Forums » Topic: Firing PIG queries from Java

DataStax Support Forums » Topic: Firing PIG queries from Java http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java Software, Support, and Training for Apache Cassandra en-US Sat, 18 May 2013 06:02:11 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "Firing PIG queries from Java" http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java#post-742 Wed, 23 Nov 2011 01:46:40 +0000 Anonymous 742@http://www.datastax.com/support-forums/ They all also particularly liked that the dresses were flattering on their stomachs. hati on "Firing PIG queries from Java" http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java#post-714 Fri, 18 Nov 2011 11:16:55 +0000 hati 714@http://www.datastax.com/support-forums/ Thanks. This got resolved after I worked a bit with the CLASSPATH. But now I am getting this error- java.lang.RuntimeException: Cannot instantiate:CassandraStorage at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:459) at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5594) at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:1577) at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1366) at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013) at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:800) at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1612) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1562) at org.apache.pig.PigServer.registerQuery(PigServer.java:534) at org.apache.pig.PigServer.registerQuery(PigServer.java:558) at com.ctva.batch.topx.TopXUtility.fireReport(TopXUtility.java:18) at com.ctva.batch.topx.TopXUtility.main(TopXUtility.java:31) Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve CassandraStorage using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.] at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:444) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:456) ... 12 more Here is my CLASSPATH /usr/share/brisk/brisk.jar:/usr/share/brisk/common/log4j-1.2.16.jar:/usr/share/brisk/common/slf4j-api-1.6.1.jar:/usr/share/brisk/common/slf4j-log4j12-1.6.1.jar:/etc/brisk/cassandra:/usr/share/brisk/cassandra/lib/antlr-2.7.7.jar:/usr/share/brisk/cassandra/lib/antlr-3.1.3.jar:/usr/share/brisk/cassandra/lib/antlr-runtime-3.1.3.jar:/usr/share/brisk/cassandra/lib/apache-cassandra-0.8.1-brisk1-beta2.jar:/usr/share/brisk/cassandra/lib/apache-cassandra-cql-1.0.1-20110620.040412-84.jar:/usr/share/brisk/cassandra/lib/apache-cassandra-thrift-0.8.1-brisk1-beta2.jar:/usr/share/brisk/cassandra/lib/avro-1.4.0-cassandra-1.jar:/usr/share/brisk/cassandra/lib/commons-cli-1.1.jar:/usr/share/brisk/cassandra/lib/commons-codec-1.2.jar:/usr/share/brisk/cassandra/lib/commons-collections-3.2.1.jar:/usr/share/brisk/cassandra/lib/commons-lang-2.4.jar:/usr/share/brisk/cassandra/lib/concurrentlinkedhashmap-lru-1.1.jar:/usr/share/brisk/cassandra/lib/guava-r08.jar:/usr/share/brisk/cassandra/lib/high-scale-lib-1.1.1.jar:/usr/share/brisk/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/brisk/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/brisk/cassandra/lib/jamm-0.2.1.jar:/usr/share/brisk/cassandra/lib/jetty-6.1.21.jar:/usr/share/brisk/cassandra/lib/jetty-util-6.1.21.jar:/usr/share/brisk/cassandra/lib/jline-0.9.94.jar:/usr/share/brisk/cassandra/lib/joda-time-1.6.2.jar:/usr/share/brisk/cassandra/lib/json-simple-1.1.jar:/usr/share/brisk/cassandra/lib/jug-2.0.0-asl.jar:/usr/share/brisk/cassandra/lib/libthrift-0.6.0.jar:/usr/share/brisk/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/brisk/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/brisk/cassandra/lib/snakeyaml-1.6.jar:/usr/share/brisk/cassandra/lib/snappy-java-1.0.3-20110604.005740-2.jar:/usr/share/brisk/cassandra/lib/stringtemplate-3.2.jar:/usr/share/brisk/brisk.jar:/usr/share/brisk/hive/lib/hive-cassandra-handler-0.7.0-brisk1-beta2.jar:/usr/share/brisk/hadoop:/usr/share/brisk/hadoop:/etc/brisk/hadoop:/usr/share/brisk/hadoop/*.jar:/usr/share/brisk/hadoop/lib/ant-1.6.5.jar:/usr/share/brisk/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/share/brisk/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/share/brisk/hadoop/lib/commons-cli-1.2.jar:/usr/share/brisk/hadoop/lib/commons-codec-1.4.jar:/usr/share/brisk/hadoop/lib/commons-collections-3.2.1.jar:/usr/share/brisk/hadoop/lib/commons-configuration-1.6.jar:/usr/share/brisk/hadoop/lib/commons-digester-1.8.jar:/usr/share/brisk/hadoop/lib/commons-el-1.0.jar:/usr/share/brisk/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/share/brisk/hadoop/lib/commons-lang-2.4.jar:/usr/share/brisk/hadoop/lib/commons-logging-1.0.4.jar:/usr/share/brisk/hadoop/lib/commons-math-2.1.jar:/usr/share/brisk/hadoop/lib/commons-net-1.4.1.jar:/usr/share/brisk/hadoop/lib/core-3.1.1.jar:/usr/share/brisk/hadoop/lib/ftplet-api-1.0.0.jar:/usr/share/brisk/hadoop/lib/ftpserver-core-1.0.0.jar:/usr/share/brisk/hadoop/lib/ftpserver-deprecated-1.0.0-M2.jar:/usr/share/brisk/hadoop/lib/hadoop-core-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-examples-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-fairscheduler-0.20.203.1-brisk1.jar:/usr/share/brisk/hadoop/lib/hadoop-streaming-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-test-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-tools-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hsqldb-1.8.0.10.jar:/usr/share/brisk/hadoop/lib/jasper-compiler-5.5.12.jar:/usr/share/brisk/hadoop/lib/jasper-runtime-5.5.12.jar:/usr/share/brisk/hadoop/lib/jets3t-0.7.1.jar:/usr/share/brisk/hadoop/lib/jetty-6.1.26.jar:/usr/share/brisk/hadoop/lib/jetty-util-6.1.26.jar:/usr/share/brisk/hadoop/lib/jsp-2.1-6.1.14.jar:/usr/share/brisk/hadoop/lib/jsp-api-2.1-6.1.14.jar:/usr/share/brisk/hadoop/lib/junit-3.8.1.jar:/usr/share/brisk/hadoop/lib/kfs-0.3.jar:/usr/share/brisk/hadoop/lib/mina-core-2.0.0-M5.jar:/usr/share/brisk/hadoop/lib/oro-2.0.8.jar:/usr/share/brisk/hadoop/lib/servlet-api-2.5-20081211.jar:/usr/share/brisk/hadoop/lib/servlet-api-2.5-6.1.14.jar:/usr/share/brisk/hadoop/lib/xmlenc-0.52.jar:/usr/share/brisk/pig/lib/ant-1.6.5.jar:/usr/share/brisk/pig/lib/commons-httpclient-3.1.jar:/usr/share/brisk/pig/lib/jackson-core-asl-1.4.2.jar:/usr/share/brisk/pig/lib/jetty-util-6.1.14.jar:/usr/share/brisk/pig/lib/paranamer-2.2.jar:/usr/share/brisk/pig/lib/xmlenc-0.52.jar:/usr/share/brisk/pig/lib/asm-3.2.jar:/usr/share/brisk/pig/lib/commons-lang-2.5.jar:/usr/share/brisk/pig/lib/jackson-mapper-asl-1.4.2.jar:/usr/share/brisk/pig/lib/jsp-2.1-6.1.14.jar:/usr/share/brisk/pig/lib/paranamer-ant-2.2.jar:/usr/share/brisk/pig/lib/avro-1.3.2.jar:/usr/share/brisk/pig/lib/commons-logging-1.0.4.jar:/usr/share/brisk/pig/lib/jasper-compiler-5.5.12.jar:/usr/share/brisk/pig/lib/jsp-api-2.1-6.1.14.jar:/usr/share/brisk/pig/lib/paranamer-generator-2.2.jar:/usr/share/brisk/pig/lib/commons-cli-1.2.jar:/usr/share/brisk/pig/lib/commons-net-1.4.1.jar:/usr/share/brisk/pig/lib/jasper-runtime-5.5.12.jar:/usr/share/brisk/pig/lib/junit-4.8.1.jar:/usr/share/brisk/pig/lib/pig-0.8.3.jar:/usr/share/brisk/pig/lib/commons-codec-1.4.jar:/usr/share/brisk/pig/lib/core-3.1.1.jar:/usr/share/brisk/pig/lib/jets3t-0.7.1.jar:/usr/share/brisk/pig/lib/kfs-0.3.jar:/usr/share/brisk/pig/lib/qdox-1.10.1.jar:/usr/share/brisk/pig/lib/commons-el-1.0.jar:/usr/share/brisk/pig/lib/hsqldb-1.8.0.10.jar:/usr/share/brisk/pig/lib/jetty-6.1.14.jar:/usr/share/brisk/pig/lib/oro-2.0.8.jar:/usr/share/brisk/pig/lib/servlet-api-2.5-6.1.14.jar zznate on "Firing PIG queries from Java" http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java#post-680 Wed, 16 Nov 2011 18:37:51 +0000 zznate 680@http://www.datastax.com/support-forums/ Can you print the classpath as pig begins to execute and make sure all the jars are there? This may simply be a matter of stepping on env. variables somewhere as well. Take a look at this section in DSE_HOME/bin/dse-env.sh and make sure it works with how you are invoking pig: export PIG_HOME=<code>dirname $0</code>/../resources/pig export PIG_CLASSPATH=$HADOOP_HOME/conf:$CLASSPATH # pig also needs some of hadoop's dependencies for jar in "$HADOOP_HOME"/*.jar "$HADOOP_HOME"/lib/*.jar; do PIG_CLASSPATH=$PIG_CLASSPATH:$jar done hati on "Firing PIG queries from Java" http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java#post-679 Wed, 16 Nov 2011 13:28:40 +0000 hati 679@http://www.datastax.com/support-forums/ Hi, I was trying to fire a bunch of PIG queries from a Java client as follows PigServer srvr = new PigServer(new PigContext()); srvr.registerQuery("raw = LOAD 'cassandra://testCounters/TOPX_TEST_COUNTERS' USING CassandraStorage() AS (name, columns: bag {T: tuple(counter, value)})"); srvr.registerQuery("rows = FOREACH raw....."); srvr.registerQuery("calls = FILTER rows by ...."); srvr.registerQuery("ordered = ORDER calls by ..."); srvr.registerQuery("limited = LIMIT ordered 10 ..."); srvr.store("limited","output"); I am getting INFO: Connecting to hadoop file system at: cfs:/// java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.fs.CassandraFileSystem I have added all the Brisk cassandra, pig, hadoop jars in the classpath. I believe I have to go through Brisk framework which I'm not doing in my code. Can someone please share how to do this?