hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Hadoop HDFS slow after upgrade vom 0.20 -> to 2.0
Date Tue, 19 Aug 2014 13:32:20 GMT
Hi Norbert,

Can you check the DN daemon's GC activity (you can look for the
JvmPauseMonitor messages in logs too, in 4.7)? Is it increased from
pre-upgrade, or generally seems high to you? What is your current DN
heap size?

One of the major changes between 0.20 to 2.0 is the BlockPool ID
concept for paths to blocks, which adds memory usage.

In addition, what OS (type and version) are you currently running?

On Tue, Aug 19, 2014 at 1:39 PM, norbi <norbi@rocknob.de> wrote:
> Hi List,
> we have upgraded Hadoop from our very old version 0.20 to Cloudera 4.7
> (hadoop 2.0), we are only using HDFS.
> After upgrade (no configuration changes), the hdfs seems to bee very slow.
> It needs more than 2h to copying 40GB(47 files) out of the hdfs, bevor
> upgrading it was about 1h.
> We are using 52 Datanodes with 10 discs, all connectet via 1gig.
> How can we speed up hdfs, or where can be the bottleneck?
> Norbert

Harsh J

View raw message