hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Rovner (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4162) CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
Date Tue, 10 Aug 2010 23:38:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897084#action_12897084
] 

Alex Rovner commented on HADOOP-4162:
-------------------------------------

We are using 0.20 and LZO can be used for map compression. In which case the same issue appears:

[2010-08-10 18:19:18,459] INFO  (JvmMetrics.java:71) - Initializing JVM Metrics with processName=SHUFFLE,
sessionId=
[2010-08-10 18:19:18,729] INFO  (GPLNativeCodeLoader.java:34) - Loaded native gpl library
[2010-08-10 18:19:18,739] INFO  (LzoCodec.java:65) - Successfully loaded & initialized
native-lzo library [hadoop-lzo rev fatal: Not a git repository]
[2010-08-10 18:19:18,793] INFO  (ReduceTask.java:1014) - ShuffleRamManager: MemoryLimit=358881696,
MaxSingleShuffleLimit=89720424
[2010-08-10 18:19:18,836] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,839] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,841] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,842] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,844] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,845] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,847] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,849] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,851] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,852] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,854] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,855] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,857] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,859] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,863] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,865] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,867] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,881] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,885] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,886] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,888] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,889] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,891] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,892] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,894] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,895] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,897] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,899] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,901] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,902] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,904] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,905] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,907] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,908] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,910] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,912] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,914] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,915] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,917] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,918] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,920] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,921] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,923] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,924] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,926] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,928] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,930] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,931] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,933] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,934] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,936] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,938] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,939] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,941] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,942] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,943] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,945] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,946] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,947] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,948] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,949] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,950] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,951] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,953] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,954] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,955] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,956] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,957] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,958] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,960] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,961] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,962] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,963] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,964] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,964] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,970] INFO  (ReduceTask.java:2468) - attempt_201008101538_0280_r_000000_0
Thread started: Thread for merging on-disk files
[2010-08-10 18:19:18,971] INFO  (ReduceTask.java:2473) - attempt_201008101538_0280_r_000000_0
Thread waiting: Thread for merging on-disk files


> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-4162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4162
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>            Assignee: Arun C Murthy
>         Attachments: HADOOP-4162_0_20080911.patch
>
>
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor. I investigated
the code, the reason seems to be the following:
> LzopCodec inherits from LzoCodec. The getDecompressorType() method is supposed to return
the concrete Decompressor class type the specific Codec class creates. In this case, LzopCodec
creates LzopDecompressors and should return LzopDecompressor.class. But instead, it uses the
getDecompressorType() method defined in the parent and returns LzoDecompressor.class.
> This leads to CodecPool unable to properly recycle the decompressors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message