Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of Milind.Bhandarkar@emc.com
 designates 128.222.32.20 as permitted sender)
From: <Milind.Bhandarkar@emc.com>
To: <mapreduce-dev@hadoop.apache.org>
Date: Fri, 5 Aug 2011 20:39:33 -0400
Subject: Re: Merge MR-279 (was Re: Status on MAPREDUCE-279 and plans forward)
Thread-Topic: Merge MR-279 (was Re: Status on MAPREDUCE-279 and plans
 forward)
Thread-Index: AcxT0UXeAlYe+7zSQ4e3CfFStQut+A==
Message-ID: <CA61DACA.15B4%Milind.Bhandarkar@emc.com>
In-Reply-To: 
 <CALF7aMTKFAg3gKTdipxYrnrsh7Gr5YA57ev=FjKHmHX2VL1H_Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.12.0.110505
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Is it building for you after hadoop-common mavenization in trunk ?

Hadoop common tests testHardLink and testFilterFileSystem are failing for
me on mac, and Mr279 mapreduce depends on these test jars:

[ERROR] Failed to execute goal on project yarn-api: Could not resolve
dependencies for project org.apache.hadoop:yarn-api:jar:1.0-SNAPSHOT:
Could not find artifact
org.apache.hadoop:hadoop-common:jar:tests:0.23.0-SNAPSHOT -> [Help 1]

- milind

---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)


On 8/5/11 5:24 PM, "Mahadev Konar" <mahadev@hortonworks.com> wrote:

>Hi all,
>  We are almost done with fixing findbugs issues. Once done, we plan to
>start a vote to merge MR-279 into trunk. If you plan to review, this would
>be a good time. We hope to start the vote next week sometime.
>
>thanks
>mahadev
>
>On Fri, Jul 29, 2011 at 3:14 PM, Mahadev Konar
><mahadev@hortonworks.com>wrote:
>
>> Good point! We are working on this. Hopefully should get this done
>>within a
>> couple of days. It would have been great if the Apache Jenkins builds
>>were
>> up and running. Ill work with Giri to add findbugs and rat to it as
>>soon as
>> they come back up.
>>
>> thanks
>> mahadev
>>
>> On Jul 29, 2011, at 12:09 AM, Todd Lipcon wrote:
>>
>> > Hey Mahadev,
>> >
>> > Have you been running findbugs/RAT on the branch? Other test-patch
>>like
>> > things?
>> >
>> > -Todd
>> >
>> > On Fri, Jul 29, 2011 at 12:06 AM, Mahadev Konar
>><mahadev@hortonworks.com
>> >wrote:
>> >
>> >> HI all,
>> >> For those working on MR-279.
>> >>
>> >> I just committed a patch to MR-279 to make Next gen MR  work with
>> >> trunk/common and trunk/hdfs. With the commit Next gen MR next would
>>no
>> >> longer work with the common/hdfs in MR-279 branch.
>> >>
>> >> We are now all set to merge Next gen MR to trunk!
>> >>
>> >> thanks
>> >> mahadev
>> >>
>> >> On Wed, Jul 27, 2011 at 9:46 AM, Mahadev Konar
>><mahadev@hortonworks.com
>> >
>> >> wrote:
>> >>> Just an update.
>> >>>
>> >>> Some good news. I have a patch for MR-279 to start working with
>> >> trunk/common and trunk/hdfs (some minor fixes). I am facing some
>>issues
>> with
>> >> secuirty which should be sorted out by EOD today.
>> >>> We should be ready to start the process of merging soon.
>> >>>
>> >>> thanks
>> >>> mahadev
>> >>>
>> >>> On Jul 12, 2011, at 11:52 PM, Arun C Murthy wrote:
>> >>>
>> >>>> On Jun 10, 2011, at 11:44 PM, Arun C Murthy wrote:
>> >>>>
>> >>>>> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and
>>MR-279.
>> >>>>> This is a very important patch by Tom which allows to submit jobs
>>to
>> >>>>> both JT and the new RM - thanks Tom!
>> >>>>> 2. Check-in the 'yarn' subfolder which is just the ResourceManager
>> and
>> >>>>> NodeManager i.e. compute fabric to trunk.
>> >>>>> 3. Rework the src tree in trunk to look like the MR-279 branch to
>> >>>>> ensure no changes to trunk are lost - this will ensure separation
>> >>>>> between the 'classic' framework (JT/TT) and the MR runtime
>> (map/reduce/
>> >>>>> sort/shuffle etc.)
>> >>>>> 4. Re-play the 4-5 patches needed to the MR runtime from the
>>MR-279
>> >>>>> branch.
>> >>>>> 5. Change build scripts to reflect the new reality.
>> >>>>
>> >>>> Hello,
>> >>>>
>> >>>> Another update...
>> >>>>
>> >>>> I've been working on 'step 4' outlined above.
>> >>>>
>> >>>> Currently the MR 'runtime' in trunk is nearly identical to the
>>MR-279
>> >> (including MAPREDUCE-2400, thanks Tom!) but for the following 4 jiras
>> for
>> >> which I'm requesting some help from the named folks to port to trunk:
>> >>>>
>> >>>> $ git show 99ab6d89acc2fc722489c317be796c15c3975284
>> >>>> Vinod - MAPREDUCE-2681. Added env vars (stderr/stdout) for hadoop
>> pipes
>> >>>>
>> >>>> $ git show 39a446c12a60041a2740d1a1b99e362a7ea1c8a5
>> >>>> Robert Evans - MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID
>> >>>>
>> >>>> $ git show 5ece56360977444ca5799dc06c581e215ff3e10b
>> >>>> Robert Evans - MAPREDUCE-2539. Fixed NPE in getMapTaskReports in
>> >> JobClient
>> >>>>
>> >>>> $ git show 6bddb76f515c3fc178078b688b8d3198b496dba3
>> >>>> Sid_S - MR-2365 in-lieu
>> >>>>
>> >>>> We also plan to port HADOOP-6929 to MR-279 (Mahadev).
>> >>>>
>> >>>> At the end of this we should have both trunk and MR-279 in sync for
>> the
>> >> MR runtime. Hopefully we can get to this by end of this week
>> >> (optimistically).
>> >>>>
>> >>>> ----
>> >>>>
>> >>>> Once we achieve this we are done with step 1 and step 4, I plan to
>>go
>> >> with the rest of the steps outlined above.
>> >>>>
>> >>>> Thus, in an ideal scenario I should be able to merge MR-279 into
>>trunk
>> >> by end of next week!
>> >>>>
>> >>>> We have already have been fortunate to receive several
>>contributions
>> >> from people not sitting in my aisle and we'd love more! *smile*
>> >>>>
>> >>>> I'd love to have more feedback and review bandwidth. Please take a
>> look
>> >> now - I'm hoping we can get this into trunk soon!
>> >>>>
>> >>>> thanks,
>> >>>> Arun
>> >>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>>
>>