DataStax Support Forums » Topic: Cassandra File System - Fixed Width Text File

DataStax Support Forums » Topic: Cassandra File System - Fixed Width Text File - Hive Integration Issue http://www.datastax.com/support-forums/topic/cassandra-file-system-fixed-width-text-file-hive-integration-issue Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 18:00:35 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php sam on "Cassandra File System - Fixed Width Text File - Hive Integration Issue" http://www.datastax.com/support-forums/topic/cassandra-file-system-fixed-width-text-file-hive-integration-issue#post-6659 Tue, 25 Sep 2012 19:04:33 +0000 sam 6659@http://www.datastax.com/support-forums/ I haven't been able to recreate the problems with the JDBC driver on either DSE 2.1 or 2.2. The only way I can get the behaviour you describe there is by running 'SHOW TABLES' from the hive cli, which triggers the bug I mentioned in the StackOverflow post. After that, queries with the JDBC driver error with "Table not found 'employees'". Regarding the other issue - is it possible to remove that temporary directory from the CFS using the hadoop command line tools (i.e. bin/dse hadoop fs)? I don't how the permissions came to be incorrect, but I think Hadoop should recreate them on demand tambalavanar on "Cassandra File System - Fixed Width Text File - Hive Integration Issue" http://www.datastax.com/support-forums/topic/cassandra-file-system-fixed-width-text-file-hive-integration-issue#post-6656 Tue, 25 Sep 2012 17:33:22 +0000 tambalavanar 6656@http://www.datastax.com/support-forums/ I'm trying to read a fixed width text file stored in Cassandra File System (CFS) using Hive. I'm able to query the file when I run from hive client. However, when I try to run from Hadoop Hive JDBC, It says table is not available or bad connection. Below are the steps I followed. <blockquote>Input file (employees.dat):</blockquote> <code> 2736Ambalavanar Thirugnanam BNYM-EAG 2005-05-091982-12-18 2737Anand Jeyamani BNYM-AST 2005-05-091984-07-12 3123Muthukumar Rajendran BNYM-EES 2009-08-121988-02-23 </code> <blockquote>Starting Hive Client</blockquote> <code>bash-3.2# dse hive; Logging initialized using configuration in file:/etc/dse/hive/hive-log4j.properties Hive history file=/tmp/root/hive_job_log_root_201209250900_157600446.txt hive> use HiveDB; OK Time taken: 1.149 seconds </code> <blockquote>Creating Hive External Table pointing to fixed width format text file</blockquote> <code> hive> CREATE EXTERNAL TABLE employees (empid STRING, firstname STRING, lastname STRING, dept STRING, dateofjoining STRING, dateofbirth STRING) > ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' > WITH SERDEPROPERTIES ("input.regex" = "(.{4})(.{25})(.{25})(.{15})(.{10})(.{10}).*" ) > LOCATION 'cfs://hostname:9160/folder/'; OK Time taken: 0.524 seconds </code> <blockquote>Do a select * from table.</blockquote> <code> hive> select * from employees; OK 2736 Ambalavanar Thirugnanam BNYM-EAG 2005-05-09 1982-12-18 2737 Anand Jeyamani BNYM-AST 2005-05-09 1984-07-12 3123 Muthukumar Rajendran BNYM-EES 2009-08-12 1988-02-23 Time taken: 0.698 seconds </code> <blockquote>Do a select with specific fields from hive table throws permission error (first issue)</blockquote> <code> hive> select empid, firstname from employees; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator java.io.IOException: The ownership/permissions on the staging directory cfs:/tmp/hadoop-root/mapred/staging/root/.staging is not as expected. It is owned by root and permissions are rwxrwxrwx. The directory must be owned by the submitter root or by root and permissions must be rwx------ at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:452) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Job Submission failed with exception 'java.io.IOException(The ownership/permissions on the staging directory cfs:/tmp/hadoop-root/mapred/staging/root/.staging is not as expected. It is owned by root and permissions are rwxrwxrwx. The directory must be owned by the submitter root or by root and permissions must be rwx------)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask </code> The second issue is, when I try to run the select * query from JDBC Hive driver (outside of dse/cassandra nodes), It says the table employees is not available. The external table created acts like a temporary table and it does not get persisted. When I use 'hive> show tables;', the employees table is not listed. Can anyone please help me figure out the problem?