Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.214.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABxyi2Rgt5zpqChh-BvPEG4kE3KY3o1vB3rV0bVB+06Ys37QfA@mail.gmail.com>
References: 
 <CABxyi2Rgt5zpqChh-BvPEG4kE3KY3o1vB3rV0bVB+06Ys37QfA@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Sat, 8 Sep 2012 08:53:16 +0530
Message-ID: 
 <CAOcnVr0euztHEGCuF7ZDqeooEhXD8kyf0k_FWEtydr+u_ke7CA@mail.gmail.com>
Subject: Re: Job Controller for MapReduce task assignment
To: user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey John,

Here's how MR works, to speak simply:

- Job.submit() is called.
- Job's InputFormat#getSplits() is called, its result serialized and
shipped across, along with other job artifacts such as jars, etc., to
the configured FS, for the JobTracker or the MR2 ApplicationMaster for
use.
- The splits info contains locality hints that the scheduler then uses
to assign a host's slot or resources to, depending also on
availability/requested resources (hence, a 'hint', not strict).

The first two are client-end (controllable), the last is dependent on
the scheduler you've put in use (Fifo/Capacity/Fair) or have
implemented (Custom).

I'm unclear on what exactly you ask, but I think you may want to start
by reading the JobSubmitter class and go around from there.

Does this help?

On Fri, Sep 7, 2012 at 1:24 PM, John Cuffney <cuffneyj@gmail.com> wrote:
> Hey,
>
> Which class handles the top level partitioning for MapReduce?  It's possible
> I have a misunderstanding of how this is handled, but in my view, there is a
> top level controller which kicks off the whole process; it handles
> partitioning of the input and distribution of the input segments to the
> various machines/tasks.  I have been searching through a lot of the Job
> classes, and they all seem to handle a single task, whereas it is important
> for me to perform some work at the highest level controller, if that exists.
> Any info on what I'm looking for/if I'm on the wrong track would be much
> appreciated.
>
> Thanks for the help,
> John


-- 
Harsh J