hadoop-hdfs-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a..@apache.org
Subject svn commit: r1444963 - in /hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs: CHANGES.HDFS-347.txt src/site/apt/ShortCircuitLocalReads.apt.vm
Date Mon, 11 Feb 2013 21:10:24 GMT
Author: atm
Date: Mon Feb 11 21:10:23 2013
New Revision: 1444963

URL: http://svn.apache.org/r1444963
HDFS-4453. Make a simple doc to describe the usage and design of the shortcircuit read feature.
Contributed by Colin Patrick McCabe.


Modified: hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-347.txt
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-347.txt?rev=1444963&r1=1444962&r2=1444963&view=diff
--- hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-347.txt (original)
+++ hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-347.txt Mon
Feb 11 21:10:23 2013
@@ -43,3 +43,5 @@ HDFS-4440. Avoid annoying log message wh
 HDFS-4473. Don't create domain socket unless we need it. (Colin Patrick McCabe via atm)
 HDFS-4485. DN should chmod socket path a+w. (Colin Patrick McCabe via atm)
+HDFS-4453. Make a simple doc to describe the usage and design of the shortcircuit read feature.
(Colin Patrick McCabe via atm)

Added: hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm?rev=1444963&view=auto
--- hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
+++ hadoop/common/branches/HDFS-347/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
Mon Feb 11 21:10:23 2013
@@ -0,0 +1,68 @@
+~~ Licensed under the Apache License, Version 2.0 (the "License");
+~~ you may not use this file except in compliance with the License.
+~~ You may obtain a copy of the License at
+~~   http://www.apache.org/licenses/LICENSE-2.0
+~~ Unless required by applicable law or agreed to in writing, software
+~~ distributed under the License is distributed on an "AS IS" BASIS,
+~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+~~ See the License for the specific language governing permissions and
+~~ limitations under the License. See accompanying LICENSE file.
+  ---
+  Hadoop Distributed File System-${project.version} - Short-Circuit Local Reads
+  ---
+  ---
+  ${maven.build.timestamp}
+HDFS Short-Circuit Local Reads
+  \[ {{{./index.html}Go Back}} \]
+* {Background}
+  In <<<HDFS>>>, reads normally go through the <<<DataNode>>>.
 Thus, when the
+  client asks the <<<DataNode>>> to read a file, the <<<DataNode>>>
reads that
+  file off of the disk and sends the data to the client over a TCP socket.
+  So-called "short-circuit" reads bypass the <<<DataNode>>>, allowing the
+  to read the file directly.  Obviously, this is only possible in cases where
+  the client is co-located with the data.  Short-circuit reads provide a
+  substantial performance boost to many applications.
+* {Configuration}
+  To configure short-circuit local reads, you will need to enable
+  <<<libhadoop.so>>>.  See
+  {{{../hadoop-common/NativeLibraries.html}Native
+  Libraries}} for details on enabling this library.
+  Short-circuit reads make use of a UNIX domain socket.  This is a special path
+  in the filesystem that allows the client and the DataNodes to communicate.
+  You will need to set a path to this socket.  The DataNode needs to be able to
+  create this path.  On the other hand, it should not be possible for any user
+  except the hdfs user or root to create this path.  For this reason, paths
+  under <<</var/run>>> or <<</var/lib>>> are often used.
+  Short-circuit local reads need to be configured on both the <<<DataNode>>>
+  and the client.
+* {Example Configuration}
+  Here is an example configuration.
+  <property>
+    <name>dfs.client.read.shortcircuit</name>
+    <value>true</value>
+  </property>
+  <property>
+    <name>dfs.domain.socket.path</name>
+    <value>/var/lib/hadoop-hdfs/dn_socket</value>
+  </property>

View raw message