hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahadev Konar <maha...@hortonworks.com>
Subject Re: Merge MR-279 (was Re: Status on MAPREDUCE-279 and plans forward)
Date Sat, 06 Aug 2011 02:56:24 GMT
You can skip running the tests for installing the common jars.

mvn clean install -DskipTests.


thanks
mahadev

On Fri, Aug 5, 2011 at 5:39 PM, <Milind.Bhandarkar@emc.com> wrote:

> Is it building for you after hadoop-common mavenization in trunk ?
>
> Hadoop common tests testHardLink and testFilterFileSystem are failing for
> me on mac, and Mr279 mapreduce depends on these test jars:
>
> [ERROR] Failed to execute goal on project yarn-api: Could not resolve
> dependencies for project org.apache.hadoop:yarn-api:jar:1.0-SNAPSHOT:
> Could not find artifact
> org.apache.hadoop:hadoop-common:jar:tests:0.23.0-SNAPSHOT -> [Help 1]
>
> - milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
>
>
> On 8/5/11 5:24 PM, "Mahadev Konar" <mahadev@hortonworks.com> wrote:
>
> >Hi all,
> >  We are almost done with fixing findbugs issues. Once done, we plan to
> >start a vote to merge MR-279 into trunk. If you plan to review, this would
> >be a good time. We hope to start the vote next week sometime.
> >
> >thanks
> >mahadev
> >
> >On Fri, Jul 29, 2011 at 3:14 PM, Mahadev Konar
> ><mahadev@hortonworks.com>wrote:
> >
> >> Good point! We are working on this. Hopefully should get this done
> >>within a
> >> couple of days. It would have been great if the Apache Jenkins builds
> >>were
> >> up and running. Ill work with Giri to add findbugs and rat to it as
> >>soon as
> >> they come back up.
> >>
> >> thanks
> >> mahadev
> >>
> >> On Jul 29, 2011, at 12:09 AM, Todd Lipcon wrote:
> >>
> >> > Hey Mahadev,
> >> >
> >> > Have you been running findbugs/RAT on the branch? Other test-patch
> >>like
> >> > things?
> >> >
> >> > -Todd
> >> >
> >> > On Fri, Jul 29, 2011 at 12:06 AM, Mahadev Konar
> >><mahadev@hortonworks.com
> >> >wrote:
> >> >
> >> >> HI all,
> >> >> For those working on MR-279.
> >> >>
> >> >> I just committed a patch to MR-279 to make Next gen MR  work with
> >> >> trunk/common and trunk/hdfs. With the commit Next gen MR next would
> >>no
> >> >> longer work with the common/hdfs in MR-279 branch.
> >> >>
> >> >> We are now all set to merge Next gen MR to trunk!
> >> >>
> >> >> thanks
> >> >> mahadev
> >> >>
> >> >> On Wed, Jul 27, 2011 at 9:46 AM, Mahadev Konar
> >><mahadev@hortonworks.com
> >> >
> >> >> wrote:
> >> >>> Just an update.
> >> >>>
> >> >>> Some good news. I have a patch for MR-279 to start working with
> >> >> trunk/common and trunk/hdfs (some minor fixes). I am facing some
> >>issues
> >> with
> >> >> secuirty which should be sorted out by EOD today.
> >> >>> We should be ready to start the process of merging soon.
> >> >>>
> >> >>> thanks
> >> >>> mahadev
> >> >>>
> >> >>> On Jul 12, 2011, at 11:52 PM, Arun C Murthy wrote:
> >> >>>
> >> >>>> On Jun 10, 2011, at 11:44 PM, Arun C Murthy wrote:
> >> >>>>
> >> >>>>> 1. I'd like to get MAPREDUCE-2400 committed to both trunk
and
> >>MR-279.
> >> >>>>> This is a very important patch by Tom which allows to submit
jobs
> >>to
> >> >>>>> both JT and the new RM - thanks Tom!
> >> >>>>> 2. Check-in the 'yarn' subfolder which is just the ResourceManager
> >> and
> >> >>>>> NodeManager i.e. compute fabric to trunk.
> >> >>>>> 3. Rework the src tree in trunk to look like the MR-279
branch to
> >> >>>>> ensure no changes to trunk are lost - this will ensure
separation
> >> >>>>> between the 'classic' framework (JT/TT) and the MR runtime
> >> (map/reduce/
> >> >>>>> sort/shuffle etc.)
> >> >>>>> 4. Re-play the 4-5 patches needed to the MR runtime from
the
> >>MR-279
> >> >>>>> branch.
> >> >>>>> 5. Change build scripts to reflect the new reality.
> >> >>>>
> >> >>>> Hello,
> >> >>>>
> >> >>>> Another update...
> >> >>>>
> >> >>>> I've been working on 'step 4' outlined above.
> >> >>>>
> >> >>>> Currently the MR 'runtime' in trunk is nearly identical to
the
> >>MR-279
> >> >> (including MAPREDUCE-2400, thanks Tom!) but for the following 4 jiras
> >> for
> >> >> which I'm requesting some help from the named folks to port to trunk:
> >> >>>>
> >> >>>> $ git show 99ab6d89acc2fc722489c317be796c15c3975284
> >> >>>> Vinod - MAPREDUCE-2681. Added env vars (stderr/stdout) for
hadoop
> >> pipes
> >> >>>>
> >> >>>> $ git show 39a446c12a60041a2740d1a1b99e362a7ea1c8a5
> >> >>>> Robert Evans - MAPREDUCE-2531. Fixed jobcontrol to downgrade
JobID
> >> >>>>
> >> >>>> $ git show 5ece56360977444ca5799dc06c581e215ff3e10b
> >> >>>> Robert Evans - MAPREDUCE-2539. Fixed NPE in getMapTaskReports
in
> >> >> JobClient
> >> >>>>
> >> >>>> $ git show 6bddb76f515c3fc178078b688b8d3198b496dba3
> >> >>>> Sid_S - MR-2365 in-lieu
> >> >>>>
> >> >>>> We also plan to port HADOOP-6929 to MR-279 (Mahadev).
> >> >>>>
> >> >>>> At the end of this we should have both trunk and MR-279 in
sync for
> >> the
> >> >> MR runtime. Hopefully we can get to this by end of this week
> >> >> (optimistically).
> >> >>>>
> >> >>>> ----
> >> >>>>
> >> >>>> Once we achieve this we are done with step 1 and step 4, I
plan to
> >>go
> >> >> with the rest of the steps outlined above.
> >> >>>>
> >> >>>> Thus, in an ideal scenario I should be able to merge MR-279
into
> >>trunk
> >> >> by end of next week!
> >> >>>>
> >> >>>> We have already have been fortunate to receive several
> >>contributions
> >> >> from people not sitting in my aisle and we'd love more! *smile*
> >> >>>>
> >> >>>> I'd love to have more feedback and review bandwidth. Please
take a
> >> look
> >> >> now - I'm hoping we can get this into trunk soon!
> >> >>>>
> >> >>>> thanks,
> >> >>>> Arun
> >> >>>
> >> >>>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Todd Lipcon
> >> > Software Engineer, Cloudera
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message