hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlos Valiente (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4707) Improvements to Hadoop Thrift bindings
Date Tue, 10 Feb 2009 23:05:59 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Carlos Valiente updated HADOOP-4707:

    Attachment: libthrift.jar

Patch HADOOP-4707.diff removes the need for an extra Thrift server
process by creating Thrift server instances on the namenode and
datanode processes. It also allows direct reads and writes to datanode

The Thrift namenode server instance implements the Thrift service
'Namenode' defined in src/thrift/hdfs.thrift. It is created by calling
in org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(),
and acts as a facade to org.apache.hadoop.hdfs.protocol.ClientProtocol.

The Thrift datanode server instance implements the Thrift service
'Datanode', defined as well in src/thrift/hdfs.thrift. It is created
in org.apache.hadoop.hdfs.datanode.DataNode.startDatanode().

In order to read data from a file, Thrift clients request a list
of blocks with Namenode.getBlocks(path, offset, length), and then
call Datanode.readBlock() on the appropriate datanode servers
for each block in the returned list. The Thrift datanode server
instance then opens a local socket to the datanode server via

Data writes are implemented in a similar way: Thrift
clients call Namenode.addBlock(path) on the Thrift
namenode server, and then call DataNode.writeBlock() on
as many Thrift Datanode servers as they wish. The Thrift
datanode servers write the block to their local storage using
and then inform the namenode about the new block using
so that full block replication is eventually achieved when the next
block report is processed by the namenode.

Namenode.addBlock() must be called on files opened by Namenode.create()
or Namenode.append(). This last call might return a Block object
representing the last partial block of an existing file, which is
currently ignored because I don't know how to handle a write to
that block (FSDataset.writeToBlock() complains about that block
being valid).

Since Datanode.readBlock() and Datanode.writeBlock() expect data blobs
of Thrift type 'binary', and that type translates to Java's byte[],
data reads and writes are limited to 2**31 -1 bytes (whereas Hadoop
blocks may be much larger, since their length is measured in longs).

The following entries in hdfs-default.xml define the locations for
the Thrift servers:

    * dfs.thrift.address, set by default to ''

    * dfs.thrift.datanode.address, set by default to ''.

The following entries limit the number and lifetime of Thrift server

    * dfs.thrift.threads.min, set by default to 5

    * dfs.thrift.threads.max, set by default to 20,

    * dfs.thrift.timeout, set by default to 60 seconds.

Thrift namenode and datanode servers try to obtain the identity of
the client by calling org.apache.hadoop.net.NetUtils.getRemoteUser(),
which implements the IDENT protocol defined by RFC 1413. If that call
fails, the value returned by security.UnixUserGroupInformation.login()
is used instead.

Perl and Python higher-level APIs for read and write functionality are
also included under src/thrift, together with test suites. Perhaps
it makes more sense to release that code independently from Hadoop,
so that they might be updated more frequently. The Perl and Python
test suites may be run with the following ant target:

    $ ant -Dcompile.thrift=true test-thrift

You may also need to pass the location of Thrift's Python library
modules, since Thrift does not install them under Python's lib
directory by default. For instance, if your Python bindings for
Thrift are installed under /opt/thrift/lib/python2.x/site-packages,
then you should do the following:

    $ ant -Dcompile.thrift=true \
	  -Dthrift.pythonpath=/opt/thrift/lib/python2.x/site-packages \

Perl and Python tests for chmod() functionality show that write
operations succeed for read-only files. I guess that's because all
tests are run as HDFS superuser.

Perl and Python high-level APIs call Namenode.addBlock() on each
invocation of Datanode.writeBlock(), so it is possible to create
files with many blocks of smaller size.

I have used Thrift version 20080411-r743112 from Thrift's
Subversion repository, in order to include the resolution of
THRIFT-193 (http://issues.apache.org/jira/browse/THRIFT-193),
which fixes Perl package generation, and THRIFT-249
(https://issues.apache.org/jira/browse/THRIFT-249), which emits Javadoc
comments for Java classes. Also attached is the libthrift.jar from that
build, which is needed at run-time by the Thrift server instances. The
patch also includes Thrift stubs for all languages supported by Thrift,
very much like the stuff under src/contrib/thiftfs.

> Improvements to Hadoop Thrift bindings
> --------------------------------------
>                 Key: HADOOP-4707
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4707
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/thiftfs
>    Affects Versions: 0.20.0
>         Environment: Tested under Linux x86-64
>            Reporter: Carlos Valiente
>            Priority: Minor
>         Attachments: all.diff, BlockManager.java, build_xml.diff, DefaultBlockManager.java,
DFSBlockManager.java, gen.diff, HADOOP-4707.diff, hadoopfs_thrift.diff, hadoopthriftapi.jar,
HadoopThriftServer.java, HadoopThriftServer_java.diff, hdfs.py, hdfs_py_venky.diff, libthrift.jar,
> I have made the following changes to hadoopfs.thrift:
> #  Added namespaces for Python, Perl and C++.
> # Renamed parameters and struct members to camelCase versions to keep them consistent
(in particular FileStatus{blockReplication,blockSize} vs FileStatus.{block_replication,blocksize}).
> # Renamed ThriftHadoopFileSystem to FileSystem. From the perspective of a Perl/Python/C++
user, 1) it is already clear that we're using Thrift, and 2) the fact that we're dealing with
Hadoop is already explicit in the namespace.  The usage of generated code is more compact
and (in my opinion) clearer:
> {quote}
>         *Perl*:
>         use HadoopFS;
>         my $client = HadoopFS::FileSystemClient->new(..);
>          _instead of:_
>         my $client = HadoopFS::ThriftHadoopFileSystemClient->new(..);
>         *Python*:
>         from hadoopfs import FileSystem
>         client = FileSystem.Client(..)
>         _instead of_
>         from hadoopfs import ThriftHadoopFileSystem
>         client = ThriftHadoopFileSystem.Client(..)
>         (See also the attached diff [^scripts_hdfs_py.diff] for the
>          new version of 'scripts/hdfs.py').
>         *C++*:
>         hadoopfs::FileSystemClient client(..);
>          _instead of_:
>         hadoopfs::ThriftHadoopFileSystemClient client(..);
> {quote}
> # Renamed ThriftHandle to FileHandle: As in 3, it is clear that we're dealing with a
Thrift object, and its purpose (to act as a handle for file operations) is clearer.
> # Renamed ThriftIOException to IOException, to keep it simpler, and consistent with MalformedInputException.
> # Added explicit version tags to fields of ThriftHandle/FileHandle, Pathname, MalformedInputException
and ThriftIOException/IOException, to improve compatibility of existing clients with future
versions of the interface which might add new fields to those objects (like stack traces for
the exception types, for instance).
> Those changes are reflected in the attachment [^hadoopfs_thrift.diff].
> Changes in generated Java, Python, Perl and C++ code are also attached in [^gen.diff].
They were generated by a Thrift checkout from trunk
> ([http://svn.apache.org/repos/asf/incubator/thrift/trunk/]) as of revision
> 719697, plus the following Perl-related patches:
> * [https://issues.apache.org/jira/browse/THRIFT-190]
> * [https://issues.apache.org/jira/browse/THRIFT-193]
> * [https://issues.apache.org/jira/browse/THRIFT-199]
> The Thrift jar file [^libthrift.jar] built from that Thrift checkout is also attached,
since it's needed to run the Java Thrift server.
> I have also added a new target to src/contrib/thriftfs/build.xml to build the Java bindings
needed for org.apache.hadoop.thriftfs.HadoopThriftServer.java (see attachment [^build_xml.diff]
and modified HadoopThriftServer.java to make use of the new bindings (see attachment [^HadoopThriftServer_java.diff]).
> The jar file [^lib/hadoopthriftapi.jar] is also included, although it can be regenerated
from the stuff under 'gen-java' and the new 'compile-gen' Ant target.
> The whole changeset is also included as [^all.diff].

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message