hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1432) Use Grizzly for Fetching Map Output in Shuffle
Date Sun, 10 Jun 2007 12:36:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12503168
] 

Devaraj Das commented on HADOOP-1432:
-------------------------------------

There are a few issues with the patch:
1) The port rolling for grizzly won't work as implemented in the patch. This is because the
port is bound to within a thread (mapOut.start()) and in the patch, you don't wait for the
bind to happen and so you won't discover that you couldn't bind to the port. After some experiments
(although not conclusive), I figured that it is not straightforward to implement that. 
I think the best way to handle this issue is to pass "0" as the port and let the OS decide.
Grizzly might need to be patched to return the port it bound to for the case where 0 is passed.
2) To get better perfomance, we need to have multiple selectors. This can be achieved by doing
mapOut.setSelectorReadThreadsCount(<number>);
3) Also, we need to have queues for the selectors - mapOut.setMaxQueueSizeInBytes(<number>);
4) The mapOut needs to be a daemon thread - mapOut.setDaemon(true).

Some more experimentation needs to be done to make sure we are using the most appropriate
Grizzly APIs (in terms of performance) before this patch can go in.
            

> Use Grizzly for Fetching Map Output in Shuffle
> ----------------------------------------------
>
>                 Key: HADOOP-1432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1432
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Tahir Hashmi
>            Assignee: Tahir Hashmi
>         Attachments: 1432.patch, grizzly.tgz
>
>
> As mentioned in HADOOP-1273 and references therefrom, Jetty 6 still doesn't seem to be
stable enough for use in Hadoop. Instead, we've decided to consider the usage of Grizzly Framework
[https://grizzly.dev.java.net/] for NIO based communication.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message