JobTracker problems in DSE 2.1 running on debian servers in AWS

(4 posts) (2 voices)

• Started 8 months ago by rk
• Latest reply from rk
• This topic is resolved

Tags:

No tags yet.

1. 

   rk
   Member

   I have a 4 node cluster out of which 2 are Hadoop (analytics) nodes. While trying the Portfolio demo, it fails at the hive execution with the following exception:

   java.net.ConnectException: Call to 50.17.205.200/50.17.205.200:8012 failed on connection exception: java.net.ConnectException: Connection refused
   at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
   at org.apache.hadoop.ipc.Client.call(Client.java:1071)
   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
   at org.apache.hadoop.mapred.$Proxy10.getProtocolVersion(Unknown Source)
   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
   at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:480)
   at org.apache.hadoop.mapred.JobClient.init(JobClient.java:474)
   at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457)
   at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
   at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
   at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:338)
   at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:436)
   at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:446)
   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:642)
   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
   Caused by: java.net.ConnectException: Connection refused
   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
   at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
   at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
   at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
   at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
   at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
   at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
   at org.apache.hadoop.ipc.Client.call(Client.java:1046)
   ... 27 more
   Job Submission failed with exception 'java.net.ConnectException(Call to 50.17.205.200/50.17.205.200:8012 failed on connection exception: java.net.ConnectException: Connection refused)'
   FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

   I also tried to access the JobTracker site using the url http://50.17.205.200:50030/jobtracker.jsp with no success

   I have checked the jobtracker ip by running the dsetool jobtracker command, and it returns 50.17.205.200. I also made this ip as the first seed node in the cassandra.yaml

   I am running the following hive command:

   dse hive -f /usr/share/dse-demos/portfolio_manager/10_day_loss.q

   any ideas????

   Posted 8 months ago #
2. 

   Piotr Kołaczkowski
   Administrator

   Are all nodes up? What is the output of dsetool ring?

   Posted 8 months ago #
3. 

   rk
   Member

   the output of dsetool ring is as shown below:

   10.242.159.111 Cassandra rack1 Cassandra Up Normal 40.23 MB 25.00% 0
   10.204.11.168 Analytics rack1 Analytics(tt) Up Normal 25.28 MB 25.00% 42535295865117307932921825928971026432
   10.76.174.178 Solr rack1 Search Up Normal 41.34 MB 25.00% 85070591730234615865843651857942052864
   10.194.82.63 Analytics rack1 Analytics(tt) Up Normal 22.85 MB 25.00% 127605887595351923798765477786913079296

   It is showing the AWS local ips for each of the nodes. I have been using the external IPs for these nodes in Cassandra.yaml and setting up the jobtracker ips.

   Here is the output of the dsetool listjt command:

   DC JobTracker
   -- --
   Analytics-Analytics 50.17.205.200
   ------
   Reserve-JT = 50.17.205.214

   Posted 8 months ago #
4. 

   rk
   Member

   I finally figured it out. The ip addresses in the cassandra.yaml should be the AWS local ip addresses, and not the external ips, as I was using. Once I made this switch, it resolved my JobTracker problems.

   Posted 8 months ago #

RSS feed for this topic

Reply

You must log in to post.