hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lukas Vlcek (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3665) WritableComparator newKey() fails for NullWritable
Date Mon, 30 Jun 2008 23:44:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609404#action_12609404
] 

Lukas Vlcek commented on HADOOP-3665:
-------------------------------------

The whole point is that I would like to understand how Reduce job can output a file without
any key values in it. The NullWritable seemed to be an ideal candidate for this but unfortunately
I ran into exceptions when trying it. So I made a quick and dirty fix which is not meant to
be a production ready (obviously NullWritable should not be special-cased in any way!).

On the other hand there seemed to be some questions which need to be asked and possible addressed.
One of them is that ReflectionUtils is able to call any constructor after setAccessible is
set to true but is this what we really want for singleton keys? And do we really need singleton
keys at all? (I believe the answer is positive). How about size (length) of key value? Is
it allowed to be zero? And why WritableComparato calls to newInstance method while this causes
issues with any class having non-public constructor?

These were the questions I was trying to acknowledge while learning Hadoop and trying to experiment
with some simple jobs having no Reduce output key...

> WritableComparator newKey() fails for NullWritable
> --------------------------------------------------
>
>                 Key: HADOOP-3665
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3665
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.16.0, 0.16.1, 0.16.2, 0.16.3, 0.16.4, 0.17.0
>         Environment: n/a
>            Reporter: Lukas Vlcek
>            Priority: Minor
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-3665.path
>
>
> It is not possible to use NullWritable as a key in order to suppress key value in output.
> Syndrome exception:
> Caused by: java.lang.IllegalAccessException: Class org.apache.hadoop.io.WritableComparator
can not access a member of class org.apache.hadoop.io.NullWritable with modifiers "private"
> The problem is that NullWritable is a singleton and does not provide public non-parametric
constructor. The following code in WritableComparator causes the exception: return (WritableComparable)keyClass.newInstance();
> Proposed simple solution is to use ReflectionUtils instead (it requires modification
as well).
> This issue is probably related to HADOOP-2922

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message