hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
Date Tue, 20 Sep 2011 22:47:10 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Suresh Srinivas updated HDFS-2246:

    Attachment: HDFS-2246.20s.2.txt

The patch has few performance issues when ported to 0.20-security due to the security mechanism.

# The ClientDatanodeProtocol RPC proxy is created using block token. This means an RPC proxy
is created for every block. This robs the performance gain from the patch.
# getBlockPathInfo() is authorized using block token. This makes the use of cache for block
to path info confusing. A cache could be added an entry based on one token and an access to
the cache using another token finds the cached path info.

The new patch makes the following changes:
# A client needs a valid, long lived ClientDatanodeProtocol proxy, instead of creating one
for each block access. In order to do that, kerberos auth method is used for RPC connectivity.
Since both the client and HDFS are running as the same user, this mechanism will work fine.
# In Datanode#getBlockInfo() following checks are made:
#* The user corresponding to the caller of this method must match the user the datanode is
running as.
#* When security enabled, the server allows only kerberos as the auth method.
# BlockReaderLocal allows short circuit read only for the user corresponding to the datanode.

Will run more tests in secure setup and also will do some more benchmarking in both secure
and non-secuire setup.

> Shortcut a local client reads to a Datanodes files directly
> -----------------------------------------------------------
>                 Key: HDFS-2246
>                 URL: https://issues.apache.org/jira/browse/HDFS-2246
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>         Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt,
HDFS-2246.20s.patch, localReadShortcut20-security.2patch

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message