hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haohui Mai <h...@hortonworks.com>
Subject Re: libhdfs portability
Date Mon, 28 Oct 2013 23:14:29 GMT
I believe that the WebHDFS API is your best bet for now. The current
implementation of WebHDFSClient does not reuse the HTTP connections, which
leads to a large part of the performance penalty.

You might want to implement your own version that reuses HTTP connection to
see whether it meets your performance requirements.


On Mon, Oct 28, 2013 at 3:38 PM, Kyle Sletmoe <
kyle.sletmoe@urbanrobotics.net> wrote:

> Now that Hadoop 2.2.0 is Windows compatible, is there going to be work on
> creating a portable version of libhdfs for C/C++ interaction with HDFS? I
> know I can use the WebHDFS REST API, but the data transfer rates are
> abysmally slow compared to the direct interaction via libhdfs.
> Regards,
> --
> Kyle Sletmoe
> *Urban Robotics Inc.**
> *Software Engineer
> 33 NW First Avenue, Suite 200 | Portland, OR 97209
> c: (541) 621-7516 | e: kyle.sletmoe@urbanrobotics.net
> http://www.urbanrobotics.net
> --
> *Information contained herein is subject to the Code of Federal Regulations
> Chapter 22 International Traffic in Arms Regulations. This data may not be
> resold, diverted, transferred, transshipped, made available to a foreign
> national within the United States, or otherwise disposed of in any other
> country outside of its intended destination, either in original form or
> after being incorporated through an intermediate process into other data
> without the prior written approval of the US Department of State.
>  **Penalties
> for violation include bans on defense and military work, fines and
> imprisonment.*

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message