hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Da Zheng <zhengda1...@gmail.com>
Subject Re: Hadoop use direct I/O in Linux?
Date Sun, 02 Jan 2011 20:25:00 GMT
PS, does FUSE DFS work in version 0.20.2?
I followed the instructions in http://wiki.apache.org/hadoop/MountableHDFS, but
when I run the following command:
ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
and get the error:
BUILD FAILED
/home/zhengda/hadoop-mod-0.20.2/build.xml:497: The following error occurred
while executing this line:
/home/zhengda/hadoop-mod-0.20.2/src/contrib/build.xml:30: The following error
occurred while executing this line:
/home/zhengda/hadoop-mod-0.20.2/src/contrib/fuse-dfs/build.xml:37: libhdfs.so
does not exist: /home/zhengda/hadoop-mod-0.20.2/build/libhdfs/libhdfs.so. Please
check flags -Dlibhdfs=1 -Dfusedfs=1 are set or first try ant compile-libhdfs
-Dlibhdfs=1

tried ant compile-libhdfs -Dlibhdfs=1
BUILD FAILED
Target "compile-libhdfs" does not exist in the project "Hadoop".

Best,
Da


On 1/2/11 3:01 PM, Da Zheng wrote:
> Hello,
> 
> direct IO can make huge performance difference, especially when Atom processors
> are used. but as far as I know, hadoop doesn't enable direct IO of Linux. Does
> anyone know any unofficial versions were developed to use direct IO?
> 
> I googled it, and found FUSE provides an option for direct IO. If I use FUSE DFS
> and enable direct IO, will I get what I want? i.e., when I write data to HDFS,
> the data is written to the disk directly (no caching by any file systems)? or
> this direct IO option only allows me to bypass the caching in FUSE and the data
> is still cached by the underlying FS?
> 
> Best,
> Da


Mime
View raw message