hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject some ideas for QJM and NFS
Date Mon, 18 Feb 2013 06:36:59 GMT

HarshJ is a good guy, I've seen this JIRA:

I have a test cluster hadoop-1.0.4, I've upgrade to hadoop-2.0.3-alpha. mu
cluster is very small, four nodes totally.

then I did some test on the original Hadoop and new Hadoop, the testing is
very simple: I have a data file with 450MB, I just put it on the HDFS.

block size: 128MB, replica: 2

the following is the result:

[root@webdm test]# ll testspeed.tar.gz
-rw-r--r-- 1 root root 452M Feb 18 13:54 testspeed.tar.gz
[root@webdm test]#

//On the hadoop-1.0.4
[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hadoop dfs -put
testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
Warning: $HADOOP_HOME is deprecated.

//On the hadoop-2.0.3-alpha with QJM
[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put testspeed.tar.gz
/ ; date +%Y-%m-%d_%H:%M:%S
13/02/18 14:13:30 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable

I do think QJM HA feature affect the performance, because each writer from
QJM, it will do: fence old writer; sync in-progress log; start new log
segment; then write. only if writer received a successful response from a
quorum of JNs, writer finished for this time.

But NFS HA just write log segment in the local and NFS, when it receive
successful response from NFS, it finished this time.

So, I just suggest we always keep these two HA features in future, even in
the stable release. which one should be used, which depends on yourself
based on your infrastructure.


View raw message