accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser" <josh.el...@gmail.com>
Subject Re: Review Request 19716: ACCUMULO-2564 Backport changes to unify Hadoop 1/2
Date Thu, 27 Mar 2014 16:58:29 GMT


> On March 27, 2014, 4:31 p.m., Sean Busbey wrote:
> > I'm presuming we don't care if unit tests built against one version of Hadoop fail
to run if executed against the libraries of the other?
> > 
> > Open issues:
> > 
> > * src/server/src/main/java/org/apache/accumulo/server/util/CountRowKeys.java uses
Counter directly, issue still exists in 1.5.2-SNAPSHOT
> > * src/server/src/main/java/org/apache/accumulo/server/test/functional/RunTests.java
uses Counter directly, issue still exists in 1.5.2-SNAPSHOT
> > * src/examples/simple/src/main/java/org/apache/accumulo/examples/simple/mapreduce/TeraSortIngest.java
uses JobContext directly, issue still exists in 1.5.2-SNAPSHOT, 1.6.0-SNAPSHOT, and master
> > * src/server/src/main/java/org/apache/accumulo/server/master/LogSort.java makes
use of TaskAttemptContext and JobContext directly, which won't work. I don't know if a fix
for Hadoop 1/2 was made before this class was removed in 1.5.0.
> > * src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/partition/RangePartitioner.java
uses JobContext directly. I don't know if a fix for Hadoop 1/2 was made before the relevant
methods were changed in a non-API-compatible way.
> > 
> > Additional needed backports:
> > 
> > * 398bc9 ctubbsii ACCUMULO-769 Deprecated and replaced 1.4.x Mapreduce APIs and
updated javadocs for all Mapreduce classes and updated referencing classes.
> > 
> >    In particular, right now you don't include the fix for AccumuloInputFormat, which
has a call to TaskAttemptContext.getConfiguration
> > 
> > * a7e159 ericnewton ACCUMULO-1809 parse short args and use reflection tricks to
update counters
> > 
> >    ContinuousMoru uses Counter directly
> > 
> >
> 
> Sean Busbey wrote:
>     Note that most of 398bc9 must not be backported, because it makes a bunch of incompatible
API changes.
> 
> Sean Busbey wrote:
>     To clarify "I'm presuming we don't care if unit tests built against one version of
Hadoop fail to run if executed against the libraries of the other?"
>     
>     right now if you, e.g. do a mvn install with one hadoop profile and then attempt
to build some non-core module with the other tests will fail with incompatible class change
errors.

Neat, I haven't seen unit tests fail like that before. Currently seeing if I can replicate
it locally in 1.6. Assuming that it's also the case there, I'm not worried about it.

I imagine fixing RunTests isn't critical (that it only affects running those tests via MR)
-- I can still run the functional tests normally so I can verify the release test reqs.


- Josh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19716/#review38739
-----------------------------------------------------------


On March 27, 2014, 6:49 a.m., Mike Drob wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19716/
> -----------------------------------------------------------
> 
> (Updated March 27, 2014, 6:49 a.m.)
> 
> 
> Review request for accumulo, Adam Fuchs, Sean Busbey, Eric Newton, and Josh Elser.
> 
> 
> Bugs: ACCUMULO-2564
>     https://issues.apache.org/jira/browse/ACCUMULO-2564
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> ACCUMULO-2564 Backport changes to unify Hadoop 1/2
> 
> This is a backport of the changes originally made for 1.5.0
> under ACCUMULO-1421 for binary compatability between Hadoop versions
> 1 and 2 with the same Accumulo artifacts. This commit is based on the
> following original work:
> 
> * c9c0d45 (Adam Fuchs)
> * d7ba6ca (Christopher Tubbs)
> * 261cf36 (Eric Newton)
> * cc3c2d8 (Eric Newton)
> 
> 
> Diffs
> -----
> 
>   src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormat.java
7cfab8b6e8f2199620c36a50ac067dee53aab6a9 
>   src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java
ed0aebf7476d8db6a968e858c9c4c892dac78bc5 
>   src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/InputFormatBase.java
a11096c3f855104db4e3b78a49f3fb039ac268ec 
>   src/server/src/main/java/org/apache/accumulo/server/Accumulo.java 184692c48ce8013067053c1d0f0dd6a7a889473a

>   src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousVerify.java
9441cf5314383705260d6fa85e69597525de421c 
> 
> Diff: https://reviews.apache.org/r/19716/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Mike Drob
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message