hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki ...@getopt.org>
Subject Re: svn commit: r446695 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/JobConf.java
Date Fri, 15 Sep 2006 22:14:58 GMT
Doug Cutting wrote:
> Andrzej Bialecki wrote:
>> Shouldn't such changes be reserved for major releases, i.e. for 0.7? 
>> Nutch relies heavily on UTF8 being the default, this change will make 
>> it more difficult to upgrade it to 0.6.2.
>
> Good question.  I think the intent was to switch as much as possible 
> from UTF8 to Text in 0.6.  Lots of things were switched, but these 
> defaults were missed.  So I was considering 0.6 the major release that 
> contains the change from UTF8 to Text in public APIs.

Hmm. Without having at least one official release where we have both 
UTF8 and Text, and the API is compatible, there will be no easy way to 
upgrade existing data. The latest release to offer this is 0.6.1, if I'm 
not mistaken - or perhaps 0.5.x, if we consider changes to SequenceFile 
format...?

If you consider users that collected terabytes of data using 0.6.1, 
there must be a way for them to upgrade this data to whatever release 
comes next. My thinking was that if we have a release that contains both 
UTF8 and Text, we could write a converter, to be included in application 
packages e.g. in Nutch in this specific release only.

Let's say I have data in SequenceFile-s and MapFile-s using 0.5.x 
formats. How would I go about converting them from UTF8 to Text? Would 
the current code read the data produced by 0.5.x code?

>
> Right now, in 0.6, the default input format is not consistent 
> (TextInputFormat now returns Text, not UTF8).  In our current monthly 
> release strategy, the .0 releases are effectively alphas, candidates 
> that sometimes are good enough to become the final release, and 
> sometimes require point releases.
>
> A consistent alternative might be to revert other places where UTF8 
> was changed to Text.
>
> http://issues.apache.org/jira/browse/HADOOP-450 (TextInputFormat)
> http://issues.apache.org/jira/browse/HADOOP-499 (contrib/streaming)
> http://issues.apache.org/jira/browse/HADOOP-460 (smallJobsBenchmark)
>
> So should we revert these in 0.6?

This looks like a lot of work ... perhaps we should just burn bridges, 
and make a 0.7.0 at this point, because it's definitely not API 
compatible with 0.6.1.

As for Nutch ... it could be upgraded to 0.6.1. On the other hand, Nutch 
is not compatible with 0.6.1 either, so perhaps it should be upgraded to 
0.7.0 (plus a suitable converter for existing data).


> I hate incompatible changes, but didn't see a way to make this change 
> compatibly, yet it seems like a good change.  What do you think?

I propose to skip 0.6.2, and go directly to 0.7.0. And I would 
appreciate any insights into the above questions  about converting old 
data ...

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Mime
View raw message