hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
Date Tue, 16 Oct 2012 23:29:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477467#comment-13477467

Todd Lipcon commented on HADOOP-6311:

bq. With respect to security: there is always a possibility for a client to open a socket
with the same name as the server would have used. This is similar to the problem with TCP/IP
sockets of a malicious program grabbing the port before the DataNode could get it (or after
the DataNode has died.)

That's why secure clusters use low (privileged) ports for the data transfer protocol.

bq. I don't think that on-disk format changes are that big of a deal for the short-circuit
pathway. We tell old clients they can't use short-circuit reads on those files, and fix new
clients to understand the new format.

Agreed, just need to make sure the "deny" pathway works and ideally some kind of version number

bq. TCP optimizations are pretty cool, but not when you run on RHEL6, as many folks do  Maybe
we should open a separate JIRA to investigate things like TCP fast open, changing TCP kernel
options, etc. might be used with Hadoop in the future. There are also certain performance
improvements we could do in the read and write paths on the DataNode, but again, that's out
of scope for this JIRA, I think.

Agreed, but my question is more this: let's assume that unix sockets for the data path are
3x as fast as local TCP sockets. If that's the case, then do we still get a big benefit from
short-circuit? I think the answer is probably yes for random read, but no for sequential.
The point about trying the "tcp friends" in future versions is just one potential way of evaluating
this without having to write all the code for a unix socket data path. If "tcp friends" is
comparable to short circuit, then unix sockets would probably also be comparable.
> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>                 Key: HADOOP-6311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6311
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: native
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Colin Patrick McCabe
>         Attachments: 6311-trunk-inprogress.txt, design.txt, HADOOP-6311.014.patch, HADOOP-6311.016.patch,
HADOOP-6311.018.patch, HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch,
HADOOP-6311.022.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in
common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message