hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Singleton <b17fly...@gmail.com>
Subject Re: is 12 minutes ok for dfs chown -R on 45000 files ?
Date Fri, 03 Oct 2008 06:33:19 GMT
Hash: SHA1

Frank Singleton wrote:
> Hi,
> Did a test on recursive chown on a fedora 9 box here (2xquad core,16Gram)
> Took about 12.5 minutes to complete for 45000 files. (hmm approx 60 files/sec)
> This was the namenode that I executed the command on
> Q1. Is this rate (60 files/sec) typical of what other folks are seeing ?
> Q2. Are there any dfs/jvm parameters I should look at to see if I can improve this
> time /home/hadoop/hadoop-0.18.1/bin/hadoop dfs -chown -R frank:frank /home/frank/proj100
> real	12m38.631s
> user	1m54.662s
> sys	0m33.124s
> time /home/hadoop/hadoop-0.18.1/bin/hadoop dfs -count /home/frank/proj100
>          220        45891         3965996260 hdfs://namenode:9000/home/frank/proj100
> real	0m1.579s
> user	0m0.686s
> sys	0m0.129s
> cheers / frank

Just to clarify, this is for when the chown will modify all files owner attributes

eg: toggle all from frank:frank to hadoop:hadoop (see below)

for chown -R from frank:frank to frank:frank , the results is only 5 or 6 seconds.

at this point , all files  under /home/frank/proj100  are frank:frank,  and the command executes
in 6 seconds or so.

[hadoop@namenode ~]$ time /home/hadoop/hadoop-0.18.1/bin/hadoop dfs -chown -R frank:frank

real	0m5.624s
user	0m6.744s
sys	0m0.402s

#now lets change all to hadoop:hadoop

[hadoop@namenode ~]$ time /home/hadoop/hadoop-0.18.1/bin/hadoop dfs -chown -R hadoop:hadoop

real	12m43.732s
user	0m53.781s
sys	0m10.655s

# now toggle back to frank:frank

[hadoop@namenode ~]$ time /home/hadoop/hadoop-0.18.1/bin/hadoop dfs -chown -R frank:frank

real	12m40.700s
user	0m45.757s
sys	0m8.173s

# now frank:frank to frank:frank

[hadoop@namenode ~]$ time /home/hadoop/hadoop-0.18.1/bin/hadoop dfs -chown -R frank:frank

real	0m5.648s
user	0m6.734s
sys	0m0.593s
[hadoop@namenode ~]$

cheers / frank

Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org


View raw message