hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Iyappan Srinivasan (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-5693) The distcp command is working slower in trunk when compared to hadoop-0.18.3
Date Fri, 17 Apr 2009 08:20:15 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Iyappan Srinivasan updated HADOOP-5693:
---------------------------------------

    Description: 
1st scenario :

Source cluster (3 slaves) is hadoop-0.18.3 and destination cluster is trunk (3 slaves).
 a) The source has 3 files in a  directory equaling 0.4 GB
 b) distcp is used to copy these contents to a  destination directory.
(bin/hadoop --config ~/cluster_dir distcp -update -delete hftp://<IPFQDN>:<port>//dir1/file1
hdfs://<IPFQDN:PORT>//copied/)
 c) It takes 34 seconds


2nd scenario :
Source cluster is trunk. and destination cluster (3 slaves) is hadoop-0.18.3
 a) The source has 3 files in a  directory equaling 0.4 GB
 b) distcp is used to copy these contents to a  destination directory.
 c) It takes 19 seconds

>From this we can conclude, In the latest trunk, distcp has considerabley slowed down (
around 78%).


  was:
1st scenario :

Source cluster (3 slaves) is hadoop-0.18.3-2486615 and destination cluster is hadoop-0.20.2678219V3
(3 slaves).
 a) The source has 3 files in a  directory equaling 0.4 GB
 b) distcp is used to copy these contents to a  destination directory.
(bin/hadoop --config ~/cluster_dir distcp -update -delete hftp://<IPFQDN>:<port>//dir1/file1
hdfs://<IPFQDN:PORT>//copied/)
 c) It takes 36 seconds


2nd scenario :
Source cluster is hadoop-0.20.2678219V3. and destination cluster (3 slaves) is hadoop-0.18.3-2486615.
 a) The source has 3 files in a  directory equaling 0.4 GB
 b) distcp is used to copy these contents to a  destination directory.
 c) It takes 20 seconds

>From this we can conclude, In the latest 0.20.2678219V3, distcp has considerabley slowed
down ( around 90%).


        Summary: The distcp command is working slower in trunk when compared to hadoop-0.18.3
 (was: The distcp command is working slower in hadoop 0.20.2678219V3 when compared to hadoop-0.18.3-2486615)

> The distcp command is working slower in trunk when compared to hadoop-0.18.3
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5693
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>         Environment: cluster is brought up with hod in both 0.18.3 and 0.20  versions
>            Reporter: Iyappan Srinivasan
>
> 1st scenario :
> Source cluster (3 slaves) is hadoop-0.18.3 and destination cluster is trunk (3 slaves).
>  a) The source has 3 files in a  directory equaling 0.4 GB
>  b) distcp is used to copy these contents to a  destination directory.
> (bin/hadoop --config ~/cluster_dir distcp -update -delete hftp://<IPFQDN>:<port>//dir1/file1
hdfs://<IPFQDN:PORT>//copied/)
>  c) It takes 34 seconds
> 2nd scenario :
> Source cluster is trunk. and destination cluster (3 slaves) is hadoop-0.18.3
>  a) The source has 3 files in a  directory equaling 0.4 GB
>  b) distcp is used to copy these contents to a  destination directory.
>  c) It takes 19 seconds
> From this we can conclude, In the latest trunk, distcp has considerabley slowed down
( around 78%).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message