hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong Tang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6588) CompressionCodecFactory throws IllegalArgumentException
Date Fri, 05 Mar 2010 22:49:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842079#action_12842079
] 

Hong Tang commented on HADOOP-6588:
-----------------------------------

bq. I am fine for keeping the current behavior, i.e. throwing an exception. However, it is
better to change the exception since it is not an IllegalArgumentException.
Makes sense, probably should let it throw ClassNotFoundException.

bq. Also, CompressionCodecFactory constructor calls getCodecClasses(..) which ultimately calls
Class.forName(..). As a consequence, Class.forName(..) will be invoked for every TextInputFormat.isSplitable(..).
It is unnecessary and expensive.

I assume you mean o.a.h.mapreduce.TextInputFormat. Not sure how to fix it in CompressionCodecFactory
though. Probably be better to cache an instance of the factory in TextInputFormat, but the
semantics is slightly different from the current code - if different conf objects are passed
down to multiple invocations of this method.

> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
>                 Key: HADOOP-6588
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6588
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Tsz Wo (Nicholas), SZE
>         Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec
not found.
>         at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
>         at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
>         at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
>         at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
>         at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
>         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
>         at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
>         at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
>         ...
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message