accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey" <s...@manvsbeard.com>
Subject Re: Review Request 19716: ACCUMULO-2564 Backport changes to unify Hadoop 1/2
Date Thu, 27 Mar 2014 17:30:39 GMT


> On March 27, 2014, 4:31 p.m., Sean Busbey wrote:
> > I'm presuming we don't care if unit tests built against one version of Hadoop fail
to run if executed against the libraries of the other?
> > 
> > Open issues:
> > 
> > * src/server/src/main/java/org/apache/accumulo/server/util/CountRowKeys.java uses
Counter directly, issue still exists in 1.5.2-SNAPSHOT
> > * src/server/src/main/java/org/apache/accumulo/server/test/functional/RunTests.java
uses Counter directly, issue still exists in 1.5.2-SNAPSHOT
> > * src/examples/simple/src/main/java/org/apache/accumulo/examples/simple/mapreduce/TeraSortIngest.java
uses JobContext directly, issue still exists in 1.5.2-SNAPSHOT, 1.6.0-SNAPSHOT, and master
> > * src/server/src/main/java/org/apache/accumulo/server/master/LogSort.java makes
use of TaskAttemptContext and JobContext directly, which won't work. I don't know if a fix
for Hadoop 1/2 was made before this class was removed in 1.5.0.
> > * src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/partition/RangePartitioner.java
uses JobContext directly. I don't know if a fix for Hadoop 1/2 was made before the relevant
methods were changed in a non-API-compatible way.
> > 
> > Additional needed backports:
> > 
> > * 398bc9 ctubbsii ACCUMULO-769 Deprecated and replaced 1.4.x Mapreduce APIs and
updated javadocs for all Mapreduce classes and updated referencing classes.
> > 
> >    In particular, right now you don't include the fix for AccumuloInputFormat, which
has a call to TaskAttemptContext.getConfiguration
> > 
> > * a7e159 ericnewton ACCUMULO-1809 parse short args and use reflection tricks to
update counters
> > 
> >    ContinuousMoru uses Counter directly
> > 
> >
> 
> Sean Busbey wrote:
>     Note that most of 398bc9 must not be backported, because it makes a bunch of incompatible
API changes.
> 
> Sean Busbey wrote:
>     To clarify "I'm presuming we don't care if unit tests built against one version of
Hadoop fail to run if executed against the libraries of the other?"
>     
>     right now if you, e.g. do a mvn install with one hadoop profile and then attempt
to build some non-core module with the other tests will fail with incompatible class change
errors.
> 
> Josh Elser wrote:
>     Neat, I haven't seen unit tests fail like that before. Currently seeing if I can
replicate it locally in 1.6. Assuming that it's also the case there, I'm not worried about
it.
>     
>     I imagine fixing RunTests isn't critical (that it only affects running those tests
via MR) -- I can still run the functional tests normally so I can verify the release test
reqs.
> 
> Mike Drob wrote:
>     For the issues that still exist in 1.5 (and further), Can we create follow on JIRAs
for that? It would certainly make the merges less painful. I'll address a7e159 and as much
of 398bc9 as makes sense in this issue.

Filed ACCUMULO-2566 for the first 3 open issues, since they persist in later branches.


- Sean


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19716/#review38739
-----------------------------------------------------------


On March 27, 2014, 6:49 a.m., Mike Drob wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19716/
> -----------------------------------------------------------
> 
> (Updated March 27, 2014, 6:49 a.m.)
> 
> 
> Review request for accumulo, Adam Fuchs, Sean Busbey, Eric Newton, and Josh Elser.
> 
> 
> Bugs: ACCUMULO-2564
>     https://issues.apache.org/jira/browse/ACCUMULO-2564
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> ACCUMULO-2564 Backport changes to unify Hadoop 1/2
> 
> This is a backport of the changes originally made for 1.5.0
> under ACCUMULO-1421 for binary compatability between Hadoop versions
> 1 and 2 with the same Accumulo artifacts. This commit is based on the
> following original work:
> 
> * c9c0d45 (Adam Fuchs)
> * d7ba6ca (Christopher Tubbs)
> * 261cf36 (Eric Newton)
> * cc3c2d8 (Eric Newton)
> 
> 
> Diffs
> -----
> 
>   src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloFileOutputFormat.java
7cfab8b6e8f2199620c36a50ac067dee53aab6a9 
>   src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java
ed0aebf7476d8db6a968e858c9c4c892dac78bc5 
>   src/core/src/main/java/org/apache/accumulo/core/client/mapreduce/InputFormatBase.java
a11096c3f855104db4e3b78a49f3fb039ac268ec 
>   src/server/src/main/java/org/apache/accumulo/server/Accumulo.java 184692c48ce8013067053c1d0f0dd6a7a889473a

>   src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousVerify.java
9441cf5314383705260d6fa85e69597525de421c 
> 
> Diff: https://reviews.apache.org/r/19716/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Mike Drob
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message