hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hequn cheng <chenghe...@gmail.com>
Subject Re: problem with HDFS caching in Hadoop 2.3
Date Sun, 09 Mar 2014 11:34:04 GMT
hi, have you solved your problem? i have the same problem. it seems that
the cache behavior has not been triggered.


2014-03-07 23:37 GMT+08:00 hwpstorage <hwpstorage@gmail.com>:

> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME     OWNER  GROUP  MODE            LIMIT  MAXTTL
> wptest1  hdfs   hdfs   rwxr-xr-x   unlimited   never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$  time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real    0m2.796s
> user    0m4.263s
> sys     0m0.203s
>
> -bash-4.1$  time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real    0m3.050s
> user    0m4.176s
> sys     0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL      REPL EXPIRY  PATH                       BYTES_NEEDED
> BYTES_CACHED  FILES_NEEDED  FILES_CACHED
>   1 wptest1      1 never   /user/hdfs/hadoop003.log
> 209715206             0             1             0
>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot

Mime
View raw message