hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From C G <parallel...@yahoo.com>
Subject Re: Problem bringing up TaskTracker on slave nodes...
Date Tue, 18 Dec 2007 21:22:17 GMT
Just to close the loop on this, and to make sure someone else doesn't have the same problem,
this turned out to be a case of cockpit error.
   
  I had mis-read the documentation concerning mapred.task.tracker.report.bindAddress and had
set it to point to the master node.  I should have left this set to 127.0.0.1 and instead
tweaked mapred.job.tracker and pointed that to the master node.  When I realized the error
of my ways I changed it and everything came up on the first try.  
   
  I could nit about the documentation needing to be a bit more clear, but I'm writing this
off as mostly a case of error due to sleep deprivation.
   
  Thanks,
  C G
  

Arun C Murthy <arunc@yahoo-inc.com> wrote:
  > 2007-12-17 14:03:59,756 ERROR org.apache.hadoop.mapred.TaskTracker: 
Can not start task tracker because java.net.BindException: Problem 
binding to master/10.2.13.1:0
> at org.apache.hadoop.ipc.Server.bind(Server.java:163)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:215)
> at org.apache.hadoop.ipc.Server.(Server.java:657)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:363)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:333)
> at 
org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:409)
> at 
org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:717)
> at 
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1880)

That tells me you need to check who is running on that port...

Arun

C G wrote:
> Sorry, forgot to paste these in.....
> 
> 
> 2007-12-17 14:03:59,371 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = master/10.2.13.1
> STARTUP_MSG: args = []
> ************************************************************/
> 2007-12-17 14:03:59,481 INFO org.mortbay.util.Credential: Checking Resource aliases
> 2007-12-17 14:03:59,531 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
> 2007-12-17 14:03:59,531 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
> 2007-12-17 14:03:59,531 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs]
> 2007-12-17 14:03:59,712 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@1f82982
> 2007-12-17 14:03:59,739 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
> 2007-12-17 14:03:59,742 INFO org.mortbay.http.SocketListener: Started SocketListener
on 0.0.0.0:50060
> 2007-12-17 14:03:59,742 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@aa37a6
> 2007-12-17 14:03:59,747 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics with processName=TaskTracker, sessionId=
> 2007-12-17 14:03:59,756 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 43913:
starting
> 2007-12-17 14:03:59,756 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 43913:
starting
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 43913:
starting
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 43913:
starting
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at:
/10.2.13.1:43913
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker master:/10.2.13.1:43913
> 2007-12-17 14:03:59,759 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 43913:
starting
> 2007-12-17 14:03:59,784 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events
fetcher for all reduce tasks on master:/10.2.13.1:43913
> 
> 2007-12-17 14:03:59,370 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = worker1/10.2.14.1
> STARTUP_MSG: args = []
> ************************************************************/
> 2007-12-17 14:03:59,481 INFO org.mortbay.util.Credential: Checking Resource aliases
> 2007-12-17 14:03:59,529 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
> 2007-12-17 14:03:59,530 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
> 2007-12-17 14:03:59,530 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs]
> 2007-12-17 14:03:59,714 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@1f82982
> 2007-12-17 14:03:59,741 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
> 2007-12-17 14:03:59,744 INFO org.mortbay.http.SocketListener: Started SocketListener
on 0.0.0.0:50060
> 2007-12-17 14:03:59,744 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@aa37a6
> 2007-12-17 14:03:59,749 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics with processName=TaskTracker, sessionId=
> 2007-12-17 14:03:59,756 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at worker1/10.2.14.1
> ************************************************************/
> 
> 
> C G 
wrote:
> Hi All:
> 
> I am migrating from a small grid to a larger one. The small grid runs fine with no issues.
On the larger grid, with nearly identical configuration files (just changing host names and
file paths), I can get dfs to run, but not all the TaskTrackers. Specifically, the task trackers
on the slave nodes fail to initialize, failing on a bind error to the master node using port
0.
> 
> The task tracker logs are below for the master (which starts up successfully) and from
one of the slaves (which fails to start). Any thoughts/comments would be most appreciated.
> 
> Note that if I log in to the worker node, and using Python do a socket.connect() to the
master on the master's port (43913 for this run) I can connect successfully. 
> 
> How do the slave nodes know what port to use when connecting? 
> 
> Any help appreciate as I am tearing out what little is left of my hair :-).
> 
> Thanks, 
> C G
> 
> 
> 
> ---------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
> 
> 
> ---------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.



       
---------------------------------
Looking for last minute shopping deals?  Find them fast with Yahoo! Search.
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message