hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 谢良 <xieli...@xiaomi.com>
Subject 答复: some ideas for QJM and NFS
Date Mon, 18 Feb 2013 07:35:33 GMT
Hi Azuryy, just want to confirm one thing, your JN did not deploy on the same machines within
DN, right ?

Regards,
Liang
________________________________
发件人: Azuryy Yu [azuryyyu@gmail.com]
发送时间: 2013年2月18日 15:22
收件人: user@hadoop.apache.org
主题: Re: some ideas for QJM and NFS

Hi,

I did it on hadoop-2.0.3-alpha without HA as following:

[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
2013-02-18_15:20:01
13/02/18 15:20:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2013-02-18_15:20:30

so the performance is a little bit better than hadoop-1.0.4.



On Mon, Feb 18, 2013 at 2:53 PM, Azuryy Yu <azuryyyu@gmail.com<mailto:azuryyyu@gmail.com>>
wrote:
Oh, yes, you are right, George. I'll probably do it in the next days.


On Mon, Feb 18, 2013 at 2:47 PM, George Datskos <george.datskos@jp.fujitsu.com<mailto:george.datskos@jp.fujitsu.com>>
wrote:
Hi Azuryy,

So you have measurements for hadoop-1.0.4 and hadoop-2.0.3+QJM, but I think you should also
measure hadoop-2.0.3 _wihout_ QJM so you can know for sure if the performance degrade is actually
related to QJM or not.


George


Hi,

HarshJ is a good guy, I've seen this JIRA: https://issues.apache.org/jira/browse/HDFS-4508

I have a test cluster hadoop-1.0.4, I've upgrade to hadoop-2.0.3-alpha. mu cluster is very
small, four nodes totally.

then I did some test on the original Hadoop and new Hadoop, the testing is very simple: I
have a data file with 450MB, I just put it on the HDFS.

block size: 128MB, replica: 2

the following is the result:

[root@webdm test]# ll testspeed.tar.gz
-rw-r--r-- 1 root root 452M Feb 18 13:54 testspeed.tar.gz
[root@webdm test]#

//On the hadoop-1.0.4
[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hadoop dfs -put testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
2013-02-18_13:54:24
Warning: $HADOOP_HOME is deprecated.
2013-02-18_13:54:58

//On the hadoop-2.0.3-alpha with QJM
[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
2013-02-18_14:13:29
13/02/18 14:13:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2013-02-18_14:14:33

I do think QJM HA feature affect the performance, because each writer from QJM, it will do:
fence old writer; sync in-progress log; start new log segment; then write. only if writer
received a successful response from a quorum of JNs, writer finished for this time.

But NFS HA just write log segment in the local and NFS, when it receive successful response
from NFS, it finished this time.

So, I just suggest we always keep these two HA features in future, even in the stable release.
which one should be used, which depends on yourself based on your infrastructure.

Thanks.



Mime
View raw message