http://www.datastax.com/support-forums/tags/pig Software, Support, and Training for Apache Cassandra en-US Tue, 21 May 2013 16:14:15 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/javalangclassnotfoundexception-dkbricsautomatonregexp-when-using-matches-in-filter-expression#post-1955 Mon, 21 May 2012 20:13:25 +0000 Anonymous 1955@http://www.datastax.com/support-forums/ <p>according to OpsCenter:<br /> Analytics 1.0.2-1<br /> Cassandra 1.0.7 </p> http://www.datastax.com/support-forums/topic/javalangclassnotfoundexception-dkbricsautomatonregexp-when-using-matches-in-filter-expression#post-1937 Fri, 18 May 2012 08:37:38 +0000 Anonymous 1937@http://www.datastax.com/support-forums/ <p>Which version of DSE are you running? </p> http://www.datastax.com/support-forums/topic/javalangclassnotfoundexception-dkbricsautomatonregexp-when-using-matches-in-filter-expression#post-1926 Wed, 16 May 2012 20:56:52 +0000 Anonymous 1926@http://www.datastax.com/support-forums/ <p>Trying to filter where the predicate looks like:</p> <p><code>p6 = filter p5 by (C2::columns::value matches &#039;c2-2&#039;)</code></p> <p>The value is from a bag (columns) of name/value tuples. </p> http://www.datastax.com/support-forums/topic/pig-error-type-mismatch-when-using-key-from-cf-in-join-or-group#post-1831 Thu, 03 May 2012 19:42:24 +0000 Anonymous 1831@http://www.datastax.com/support-forums/ <p>Unfortunately there is no work around currently available, actually you have to options: a. you would have to wait for 2.1 release with updated CassandraStorage, b. you would have to re-compile C*, include it into DSE and use org.apache.cassandra.hadoop.pig.CassandraStorage class at your own risk. </p> http://www.datastax.com/support-forums/topic/agregate-data-by-columns-in-pig#post-1828 Thu, 03 May 2012 15:28:19 +0000 Anonymous 1828@http://www.datastax.com/support-forums/ <p>Would it help to flatten the bag out first? then you can use of foreach or group to re-arrange. </p> <p>e.g.<br /> <code><br /> p1 = LOAD 'cassandra://keyspace/CF' USING CassandraStorage() AS (key:chararray, columns: bag{T: tuple(property:chararray, value:chararray)});<br /> p2 = FOREACH p1 GENERATE key, flatten (columns);<br /> </code></p> <p>You'll get a tuples of (key, property, value) and can then filter, split, group by, etc..</p> <p>Although, let me know if you're able use GROUP BY or JOIN on the key, I keep getting an error - recently created a post, but no replies yet. </p> http://www.datastax.com/support-forums/topic/agregate-data-by-columns-in-pig#post-1821 Wed, 02 May 2012 16:47:39 +0000 Anonymous 1821@http://www.datastax.com/support-forums/ <p>Probably the only way would be to add all/some of the columns to the metadata so they become separated in the Pig schema. </p> http://www.datastax.com/support-forums/topic/agregate-data-by-columns-in-pig#post-1820 Wed, 02 May 2012 16:35:12 +0000 Anonymous 1820@http://www.datastax.com/support-forums/ <p>I have data in Cassandra in following structure:</p> <pre><code>[default@data] list values; Using default limit of 100 ------------------- RowKey: 123:1333065600000 =&gt; (column=16000, value=331, timestamp=1333724446041000) =&gt; (column=76000, value=314, timestamp=1333724446042000) =&gt; (column=136000, value=333, timestamp=1333724446043000) RowKey: 123:1332979200000 =&gt; (column=6000, value=300, timestamp=1333743661692000) =&gt; (column=66000, value=302, timestamp=1333743661692001) =&gt; (column=126000, value=303, timestamp=1333743661694000) =&gt; (column=186000, value=234, timestamp=1333743661695000) =&gt; (column=246000, value=445, timestamp=1333743661696000) =&gt; (column=306000, value=331, timestamp=1333743661696001) =&gt; (column=366000, value=455, timestamp=1333743661698000) RowKey: 121:1334102400000 =&gt; (column=68608000, value=12, timestamp=1334173122715000) =&gt; (column=68668000, value=12, timestamp=1334173122715001)</code></pre> <p>In other words - number of columns can differ row per row. Now I need to do aggregation using pig of all values. I cannot change the structure. Any idea how to do that? </p> http://www.datastax.com/support-forums/topic/pig-error-type-mismatch-when-using-key-from-cf-in-join-or-group#post-1787 Fri, 27 Apr 2012 16:39:00 +0000 Anonymous 1787@http://www.datastax.com/support-forums/ <p>Using DSE 1.x<br /> I have a simple CF with UTF8 key, column, &amp; value.</p> <p>I can use the column or value in a join or group by, but if I try to use the key, I get the following error:</p> <blockquote><p> Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableBytesWritable </p></blockquote> <p>Simplest example:<br /> <code><br /> p1 = LOAD 'cassandra://keyspace/CF' USING CassandraStorage() AS (key:chararray, columns: bag{T: tuple(property:chararray, value:chararray)});<br /> p2 = group p1 by columns::property;<br /> dump p2;<br /> </code></p> <p>Works just fine, but changing the group by to:<br /> <code><br /> p2 = group p1 by key;<br /> dump p2;<br /> </code></p> <p>will fail with the type mismatch error.</p> <p>However, if I do a load, using the same schema, from a text file with lines of the form:</p> <blockquote><p> key {(c1,v1),(c2,v2),(c3,v3)} </p></blockquote> <p>I can group (or join) by the key successfully.</p> <p>Any suggestions for work around?</p> <p>Thanks,<br /> John </p> http://www.datastax.com/support-forums/topic/exception-when-running-pig-mapreduce-job-written-in-java#post-1747 Mon, 23 Apr 2012 23:38:54 +0000 Anonymous 1747@http://www.datastax.com/support-forums/ <p>You need to run with using ./bin/dse script otherwise you won't have "dse.job.tracker" variable (and probably other important ones) set. </p> http://www.datastax.com/support-forums/topic/exception-when-running-pig-mapreduce-job-written-in-java#post-1654 Tue, 17 Apr 2012 16:39:54 +0000 Anonymous 1654@http://www.datastax.com/support-forums/ <p>From the command-line on one of the nodes in the cluster. This is the command I am running:</p> <p>java -cp .:./bin:/opt/datastax/dse-2.0-1/resources/hadoop/conf/:/opt/datastax/dse-2.0-1/resources/pig/pig-0.8.3.jar:/opt/datastax/dse-2.0-1/resources/pig/lib/*:/opt/datastax/dse-2.0-1/resources/hadoop/lib/*:/opt/datastax/dse-2.0-1/resources/hadoop/*:/opt/datastax/dse-2.0-1/resources/dse/lib/*:/opt/datastax/dse-2.0-1/lib/dse-2.0-1.jar:/opt/datastax/dse-2.0-1/resources/cassandra/lib/*:/opt/datastax/dse-2.0-1/resources/cassandra/conf/:/opt/datastax/dse-2.0-1/resources/dse/conf/ idmapreduce </p> http://www.datastax.com/support-forums/topic/exception-when-running-pig-mapreduce-job-written-in-java#post-1653 Tue, 17 Apr 2012 16:26:01 +0000 tjake 1653@http://www.datastax.com/support-forums/ <p>How are you running this job? </p> http://www.datastax.com/support-forums/topic/exception-when-running-pig-mapreduce-job-written-in-java#post-1652 Tue, 17 Apr 2012 16:24:36 +0000 Anonymous 1652@http://www.datastax.com/support-forums/ <p>I am trying to run an example Pig MapReduce job written in Java against a multi-node DataStax Enterprise cluster, but I am getting an exception. The example code I am using is found here, in the "Mapreduce Mode" section: <a href="http://pig.apache.org/docs/r0.9.1/cont.html" rel="nofollow">http://pig.apache.org/docs/r0.9.1/cont.html</a>.</p> <p>The exception I am getting is:</p> <p>org.apache.pig.PigException: ERROR 1002: Unable to store alias B<br /> at org.apache.pig.PigServer.storeEx(PigServer.java:889)<br /> at org.apache.pig.PigServer.store(PigServer.java:827)<br /> at org.apache.pig.PigServer.store(PigServer.java:795)<br /> at idmapreduce.runIdQuery(idmapreduce.java:16)<br /> at idmapreduce.main(idmapreduce.java:7)<br /> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.<br /> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:403)<br /> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1209)<br /> at org.apache.pig.PigServer.storeEx(PigServer.java:885)<br /> ... 4 more<br /> Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: ${dse.job.tracker}:50020<br /> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:170)<br /> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:151)<br /> at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:2567)<br /> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:474)<br /> at org.apache.hadoop.mapred.JobClient.&lt;init&gt;(JobClient.java:457)<br /> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:122)<br /> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)<br /> ... 6 more</p> <p>I think the problem has to do with this line:</p> <p>Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: ${dse.job.tracker}:50020</p> <p>but I don't know what it means exactly. Are there some configurations I could be missing in the setup? </p> http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java#post-742 Wed, 23 Nov 2011 01:46:40 +0000 Anonymous 742@http://www.datastax.com/support-forums/ <p>They all also particularly liked that the dresses were flattering on their stomachs. </p> http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java#post-714 Fri, 18 Nov 2011 11:16:55 +0000 hati 714@http://www.datastax.com/support-forums/ <p>Thanks. This got resolved after I worked a bit with the CLASSPATH. But now I am getting this error-<br /> java.lang.RuntimeException: Cannot instantiate:CassandraStorage<br /> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:459)<br /> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5594)<br /> at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:1577)<br /> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1366)<br /> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013)<br /> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:800)<br /> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)<br /> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1612)<br /> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1562)<br /> at org.apache.pig.PigServer.registerQuery(PigServer.java:534)<br /> at org.apache.pig.PigServer.registerQuery(PigServer.java:558)<br /> at com.ctva.batch.topx.TopXUtility.fireReport(TopXUtility.java:18)<br /> at com.ctva.batch.topx.TopXUtility.main(TopXUtility.java:31)<br /> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve CassandraStorage using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]<br /> at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:444)<br /> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:456)<br /> ... 12 more</p> <p>Here is my CLASSPATH<br /> /usr/share/brisk/brisk.jar:/usr/share/brisk/common/log4j-1.2.16.jar:/usr/share/brisk/common/slf4j-api-1.6.1.jar:/usr/share/brisk/common/slf4j-log4j12-1.6.1.jar:/etc/brisk/cassandra:/usr/share/brisk/cassandra/lib/antlr-2.7.7.jar:/usr/share/brisk/cassandra/lib/antlr-3.1.3.jar:/usr/share/brisk/cassandra/lib/antlr-runtime-3.1.3.jar:/usr/share/brisk/cassandra/lib/apache-cassandra-0.8.1-brisk1-beta2.jar:/usr/share/brisk/cassandra/lib/apache-cassandra-cql-1.0.1-20110620.040412-84.jar:/usr/share/brisk/cassandra/lib/apache-cassandra-thrift-0.8.1-brisk1-beta2.jar:/usr/share/brisk/cassandra/lib/avro-1.4.0-cassandra-1.jar:/usr/share/brisk/cassandra/lib/commons-cli-1.1.jar:/usr/share/brisk/cassandra/lib/commons-codec-1.2.jar:/usr/share/brisk/cassandra/lib/commons-collections-3.2.1.jar:/usr/share/brisk/cassandra/lib/commons-lang-2.4.jar:/usr/share/brisk/cassandra/lib/concurrentlinkedhashmap-lru-1.1.jar:/usr/share/brisk/cassandra/lib/guava-r08.jar:/usr/share/brisk/cassandra/lib/high-scale-lib-1.1.1.jar:/usr/share/brisk/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/brisk/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/brisk/cassandra/lib/jamm-0.2.1.jar:/usr/share/brisk/cassandra/lib/jetty-6.1.21.jar:/usr/share/brisk/cassandra/lib/jetty-util-6.1.21.jar:/usr/share/brisk/cassandra/lib/jline-0.9.94.jar:/usr/share/brisk/cassandra/lib/joda-time-1.6.2.jar:/usr/share/brisk/cassandra/lib/json-simple-1.1.jar:/usr/share/brisk/cassandra/lib/jug-2.0.0-asl.jar:/usr/share/brisk/cassandra/lib/libthrift-0.6.0.jar:/usr/share/brisk/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/brisk/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/brisk/cassandra/lib/snakeyaml-1.6.jar:/usr/share/brisk/cassandra/lib/snappy-java-1.0.3-20110604.005740-2.jar:/usr/share/brisk/cassandra/lib/stringtemplate-3.2.jar:/usr/share/brisk/brisk.jar:/usr/share/brisk/hive/lib/hive-cassandra-handler-0.7.0-brisk1-beta2.jar:/usr/share/brisk/hadoop:/usr/share/brisk/hadoop:/etc/brisk/hadoop:/usr/share/brisk/hadoop/*.jar:/usr/share/brisk/hadoop/lib/ant-1.6.5.jar:/usr/share/brisk/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/share/brisk/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/share/brisk/hadoop/lib/commons-cli-1.2.jar:/usr/share/brisk/hadoop/lib/commons-codec-1.4.jar:/usr/share/brisk/hadoop/lib/commons-collections-3.2.1.jar:/usr/share/brisk/hadoop/lib/commons-configuration-1.6.jar:/usr/share/brisk/hadoop/lib/commons-digester-1.8.jar:/usr/share/brisk/hadoop/lib/commons-el-1.0.jar:/usr/share/brisk/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/share/brisk/hadoop/lib/commons-lang-2.4.jar:/usr/share/brisk/hadoop/lib/commons-logging-1.0.4.jar:/usr/share/brisk/hadoop/lib/commons-math-2.1.jar:/usr/share/brisk/hadoop/lib/commons-net-1.4.1.jar:/usr/share/brisk/hadoop/lib/core-3.1.1.jar:/usr/share/brisk/hadoop/lib/ftplet-api-1.0.0.jar:/usr/share/brisk/hadoop/lib/ftpserver-core-1.0.0.jar:/usr/share/brisk/hadoop/lib/ftpserver-deprecated-1.0.0-M2.jar:/usr/share/brisk/hadoop/lib/hadoop-core-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-examples-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-fairscheduler-0.20.203.1-brisk1.jar:/usr/share/brisk/hadoop/lib/hadoop-streaming-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-test-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hadoop-tools-0.20.203.1-brisk1-beta2.jar:/usr/share/brisk/hadoop/lib/hsqldb-1.8.0.10.jar:/usr/share/brisk/hadoop/lib/jasper-compiler-5.5.12.jar:/usr/share/brisk/hadoop/lib/jasper-runtime-5.5.12.jar:/usr/share/brisk/hadoop/lib/jets3t-0.7.1.jar:/usr/share/brisk/hadoop/lib/jetty-6.1.26.jar:/usr/share/brisk/hadoop/lib/jetty-util-6.1.26.jar:/usr/share/brisk/hadoop/lib/jsp-2.1-6.1.14.jar:/usr/share/brisk/hadoop/lib/jsp-api-2.1-6.1.14.jar:/usr/share/brisk/hadoop/lib/junit-3.8.1.jar:/usr/share/brisk/hadoop/lib/kfs-0.3.jar:/usr/share/brisk/hadoop/lib/mina-core-2.0.0-M5.jar:/usr/share/brisk/hadoop/lib/oro-2.0.8.jar:/usr/share/brisk/hadoop/lib/servlet-api-2.5-20081211.jar:/usr/share/brisk/hadoop/lib/servlet-api-2.5-6.1.14.jar:/usr/share/brisk/hadoop/lib/xmlenc-0.52.jar:/usr/share/brisk/pig/lib/ant-1.6.5.jar:/usr/share/brisk/pig/lib/commons-httpclient-3.1.jar:/usr/share/brisk/pig/lib/jackson-core-asl-1.4.2.jar:/usr/share/brisk/pig/lib/jetty-util-6.1.14.jar:/usr/share/brisk/pig/lib/paranamer-2.2.jar:/usr/share/brisk/pig/lib/xmlenc-0.52.jar:/usr/share/brisk/pig/lib/asm-3.2.jar:/usr/share/brisk/pig/lib/commons-lang-2.5.jar:/usr/share/brisk/pig/lib/jackson-mapper-asl-1.4.2.jar:/usr/share/brisk/pig/lib/jsp-2.1-6.1.14.jar:/usr/share/brisk/pig/lib/paranamer-ant-2.2.jar:/usr/share/brisk/pig/lib/avro-1.3.2.jar:/usr/share/brisk/pig/lib/commons-logging-1.0.4.jar:/usr/share/brisk/pig/lib/jasper-compiler-5.5.12.jar:/usr/share/brisk/pig/lib/jsp-api-2.1-6.1.14.jar:/usr/share/brisk/pig/lib/paranamer-generator-2.2.jar:/usr/share/brisk/pig/lib/commons-cli-1.2.jar:/usr/share/brisk/pig/lib/commons-net-1.4.1.jar:/usr/share/brisk/pig/lib/jasper-runtime-5.5.12.jar:/usr/share/brisk/pig/lib/junit-4.8.1.jar:/usr/share/brisk/pig/lib/pig-0.8.3.jar:/usr/share/brisk/pig/lib/commons-codec-1.4.jar:/usr/share/brisk/pig/lib/core-3.1.1.jar:/usr/share/brisk/pig/lib/jets3t-0.7.1.jar:/usr/share/brisk/pig/lib/kfs-0.3.jar:/usr/share/brisk/pig/lib/qdox-1.10.1.jar:/usr/share/brisk/pig/lib/commons-el-1.0.jar:/usr/share/brisk/pig/lib/hsqldb-1.8.0.10.jar:/usr/share/brisk/pig/lib/jetty-6.1.14.jar:/usr/share/brisk/pig/lib/oro-2.0.8.jar:/usr/share/brisk/pig/lib/servlet-api-2.5-6.1.14.jar </p> http://www.datastax.com/support-forums/topic/firing-pig-queries-from-java#post-680 Wed, 16 Nov 2011 18:37:51 +0000 zznate 680@http://www.datastax.com/support-forums/ <p>Can you print the classpath as pig begins to execute and make sure all the jars are there? This may simply be a matter of stepping on env. variables somewhere as well. </p> <p>Take a look at this section in DSE_HOME/bin/dse-env.sh and make sure it works with how you are invoking pig: </p> <p>export PIG_HOME=<code>dirname $0</code>/../resources/pig<br /> export PIG_CLASSPATH=$HADOOP_HOME/conf:$CLASSPATH<br /> # pig also needs some of hadoop's dependencies<br /> for jar in "$HADOOP_HOME"/*.jar "$HADOOP_HOME"/lib/*.jar; do<br /> PIG_CLASSPATH=$PIG_CLASSPATH:$jar<br /> done </p>