accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Tubbs (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-884) Take advantage of short circuit read for local files
Date Thu, 10 Oct 2013 00:01:42 GMT


Christopher Tubbs commented on ACCUMULO-884:

Well, the local files are created as the data node, so they would have permissions for the
data node. Unless the data node process is writing files that are world readable on the local
filesystem (I hope not, but I expect this to be controllable with umask or similar, or by
controlling read or execute permissions on the directory where the local files are stored),
it should be the case that nobody else can do local reads on those files. It was my understanding
that this is how the feature was implemented... that the client read the local files directly.

[~kturner] and I were talking and it seems there is a possibility this could be made to work,
via native libraries where the data node uses its permissions to get a file handle, and passes
that to the client via some IPC, but to the best of my knowledge this is not the way this
feature was implemented in HDFS. (And, I'm not sure what the SElinux implications would be
for such a mechanism, if that was turned on, either.)

Another way they could have implemented this is by sending through some local unix socket
or some other streamlined "local" RPC call to the data node, but I don't think they did that
either... and it would still go through the data node if it did, and I'm not sure that would
be any more performant than a regular RPC call that just happened to route within the local

> Take advantage of short circuit read for local files
> ----------------------------------------------------
>                 Key: ACCUMULO-884
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: docs
>            Reporter: Billie Rinaldi
>            Assignee: Keith Turner
> This is a new feature in hadoop 1.0.x and some versions of 0.22 and 0.23.  It allows
a client to read directly from disk instead of through a DataNode when the data is stored
locally.  Enabling it involves setting two configuration parameters, the first in hdfs-site.xml
and the second in accumulo-site.xml.  We should make sure this works with Accumulo and recommend
it in the documentation.
> - dfs.block.local-path-access.user is the key in datanode configuration to specify the
user allowed to do short circuit read.
> - is the key to enable short circuit read at the client
side configuration.
> See HDFS-2246 and for more information.

This message was sent by Atlassian JIRA

View raw message