hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Reopened: (HADOOP-16) RPC call times out while indexing map task is computing splits
Date Wed, 22 Feb 2006 20:26:42 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-16?page=all ]
     
Doug Cutting reopened HADOOP-16:
--------------------------------


I'm re-opening this & assigning it to Mike.

I think we can fix this by changing the TaskTracker to incrementally calculate things, as
I alluded above.  The TaskTracker should avoid ever doing anything that might take very long.
 RPC timeouts to the TaskTracker are very bad and must be avoided.

We can maintain a mapping of taskTracker->split, initially empty, and a queue of splits,
initially filled with all of the splits.  (If basic split-generation is too expensive, since
it calls file-length on each input file,, then we can eventually change the split API into
a generator, which incrementally enumerates splits and use that in place of this queue.) 
When a request for a task arrives from a tasktracker we can first examine the table.  If any
splits are present for the calling tracker, then we return one and remove it from the table.
 Otherwise we pop a constant number of splits (10?) off of the queue, enumerate the tasktrackers
that host each split by calling the namenode, and add entries to the table.  Then we consult
the table again.  In most cases (many more splits than tasktrackers) we should identify suitable
splits.  If we fail then we assign a randomly selected, non-local split to the tasktracker.
 Make sense?

> RPC call times out while indexing map task is computing splits
> --------------------------------------------------------------
>
>          Key: HADOOP-16
>          URL: http://issues.apache.org/jira/browse/HADOOP-16
>      Project: Hadoop
>         Type: Bug
>   Components: mapred
>     Versions: 0.1
>  Environment: MapReduce multi-computer crawl environment: 11 machines (1 master with
JobTracker/NameNode, 10 slaves with TaskTrackers/DataNodes)
>     Reporter: Chris Schneider
>     Assignee: Mike Cafarella
>      Fix For: 0.1
>  Attachments: patch.16
>
> We've been using Nutch 0.8 (MapReduce) to perform some internet crawling. Things seemed
to be going well until...
> 060129 222409 Lost tracker 'tracker_56288'
> 060129 222409 Task 'task_m_10gs5f' has been lost.
> 060129 222409 Task 'task_m_10qhzr' has been lost.
>    ........
>    ........
> 060129 222409 Task 'task_r_zggbwu' has been lost.
> 060129 222409 Task 'task_r_zh8dao' has been lost.
> 060129 222455 Server handler 8 on 8010 caught: java.net.SocketException: Socket closed
> java.net.SocketException: Socket closed
>         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>         at org.apache.nutch.ipc.Server$Handler.run(Server.java:216)
> 060129 222455 Adding task 'task_m_cia5po' to set for tracker 'tracker_56288'
> 060129 223711 Adding task 'task_m_ffv59i' to set for tracker 'tracker_25647'
> I'm hoping that someone could explain why task_m_cia5po got added to tracker_56288 after
this tracker was lost.
> The Crawl .main process died with the following output:
> 060129 221129 Indexer: adding segment: /user/crawler/crawl-20060129091444/segments/20060129200246
> Exception in thread "main" java.io.IOException: timed out waiting for response
>     at org.apache.nutch.ipc.Client.call(Client.java:296)
>     at org.apache.nutch.ipc.RPC$Invoker.invoke(RPC.java:127)
>     at $Proxy1.submitJob(Unknown Source)
>     at org.apache.nutch.mapred.JobClient.submitJob(JobClient.java:259)
>     at org.apache.nutch.mapred.JobClient.runJob(JobClient.java:288)
>     at org.apache.nutch.indexer.Indexer.index(Indexer.java:263)
>     at org.apache.nutch.crawl.Crawl.main(Crawl.java:127)
> However, it definitely seems as if the JobTracker is still waiting for the job to finish
(no failed jobs).
> Doug Cutting's response:
> The bug here is that the RPC call times out while the map task is computing splits. 
The fix is that the job tracker should not compute splits until after it has returned from
the submitJob RPC.  Please submit a bug in Jira to help remind us to fix this.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message