hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: iozone is not working with HDFS over libfuse
Date Tue, 06 Jul 2010 18:56:45 GMT
Hey Christian,

HDFS does not support the full set of POSIX semantics.  I suspect bonnie++ is exercising things
which HDFS doesn't support (such as writes to a file which is already closed or seeks during
writes).

Additionally, I would warn that, as bonnie++ is designed to test the performance of harddrives
/ network file systems / cluster file systems, you will find it doesn't capture well the performance
aspects of a distributed file system.

In other words, you are performance testing an apple with a test suite designed for oranges.

Brian

On Jul 6, 2010, at 1:14 PM, Christian Baun wrote:

> Hi,
> 
> After some time, bonnie++ exits too with this error message:
> 
> Can't write block.: Invalid argument
> Bonnie: drastic I/O error (re write(2)): Invalid argument
> [1]+  Exit 1                  bonnie++ -s 3584 -d /hdfs/ -x5 -u hadoop -q -b -m HDFS_test
> /tmp/ausgabe.txt 2>&1
> 
> This looks not so good. :-(
> 
> Best Regards,
>   Christian
> 
> 
> Am Dienstag, 6. Juli 2010 schrieb Christian Baun:
>> Hi,
>> 
>> I installed a small Hadoop-Cluster with Cloudera CDH3.
>> 
>> 2 Server are running. One is running the NameNode and DataNode. The other server
is just a DataNode.
>> A dedicated Client exists too.
>> 
>> I want to run some performance tests with bonnie++ and iozone.
>> bonnie++ works but when I start iozone, it interrups.
>> 
>> Is this a known issue? 
>> 
>> $ mount | grep hdfs
>> fuse_dfs on /hdfs type fuse.fuse_dfs (rw,nosuid,nodev,allow_other,default_permissions)
>> 
>> $ iozone -Rac -g 3584M -f /hdfs/testfile | tee -a /tmp/iozone_results.txt &
>> [1] 6283
>> hadoop@ip-10-243-13-191:/home/ubuntu$ 	Iozone: Performance Test of File I/O
>> 	        Version $Revision: 3.308 $
>> 		Compiled for 32 bit mode.
>> 		Build: linux 
>> 
>> 	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
>> 	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
>> 	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
>> 	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
>> 	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
>> 	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
>> 
>> 	Run began: Tue Jul  6 16:40:35 2010
>> 
>> 	Excel chart generation enabled
>> 	Auto Mode
>> 	Include close in write timing
>> 	Using maximum file size of 3670016 kilobytes.
>> 	Command line used: iozone -Rac -g 3584M -f /hdfs/testfile
>> 	Output is in Kbytes/sec
>> 	Time Resolution = 0.000001 seconds.
>> 	Processor cache size set to 1024 Kbytes.
>> 	Processor cache line size set to 32 bytes.
>> 	File stride size set to 17 * record size.
>>                                                            random  random    bkwd
  record   stride                                   
>>              KB  reclen   write rewrite    read    reread    read   write    read
 rewrite     read   fwrite frewrite   fread  freread
>>              64       4write: Invalid argument
>> 
>> Error writing block 0, fd= 3
>> 
>> iozone: interrupted
>> 
>> 
>> Best Regards, 
>>    Christian 
>> 
> 


Mime
View raw message