hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@alumni.cmu.edu>
Subject Re: problem with HDFS caching in Hadoop 2.3
Date Tue, 11 Mar 2014 19:55:28 GMT
On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage <hwpstorage@gmail.com> wrote:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME     OWNER  GROUP  MODE            LIMIT  MAXTTL
> wptest1  hdfs   hdfs   rwxr-xr-x   unlimited   never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$  time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real    0m2.796s
> user    0m4.263s
> sys     0m0.203s
>
> -bash-4.1$  time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real    0m3.050s
> user    0m4.176s
> sys     0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL      REPL EXPIRY  PATH                       BYTES_NEEDED
> BYTES_CACHED  FILES_NEEDED  FILES_CACHED
>   1 wptest1      1 never   /user/hdfs/hadoop003.log      209715206
> 0             1             0

If you take a look at this output, you can see that nothing is actually cached.

One way to figure out why this is is to look at the logs of the
NameNode and DataNode.  Some of the relevant logs are at DEBUG or
TRACE level, so you may need to turn up the logs.  The
CacheReplicationMonitor and FsDatasetCache classes are good places to
start.

Also be sure to check that you have set dfs.datanode.max.locked.memory.

As Andrew commented, "hadoop tail" is not a good command to use for
measuring performance, since you have a few seconds of Java startup
time, followed by any HDFS setup time, followed by reading a single
kilobyte of data.  If you want to use the shell, the simplest thing to
do is to use cat and read a large file, so that those startup costs
don't dominate the measurement.

best,
Colin


>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot

Mime
View raw message