Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.214.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAFFf+k_UgweW9iV6iOGDY-8QdJuh=cK_nFE86xOp1BvrUunRiQ@mail.gmail.com>
References: 
 <CAFFf+k_UgweW9iV6iOGDY-8QdJuh=cK_nFE86xOp1BvrUunRiQ@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Tue, 21 Aug 2012 15:09:44 +0530
Message-ID: 
 <CAOcnVr3UeHED3S6JDh7g=sAU7eH64zcrU6g4eTyz+sTh78KoUw@mail.gmail.com>
Subject: Re: Extension points available for data locality
To: user@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Tharindu,

(Am assuming you've done enough research to know that there's benefit
in what you're attempting to do.)

Locality of tasks are determined by the job's InputFormat class.
Specifically, the locality information returned by the InputSplit
objects via InputFormat#getSplits(=85) API is what the MR scheduler
looks at when trying to launch data local tasks.

You can tweak your InputFormat (the one that uses this DB as input?)
to return relevant locations based on your "DB Cluster", in order to
achieve this.

On Tue, Aug 21, 2012 at 2:36 PM, Tharindu Mathew <mccloud35@gmail.com> wrot=
e:
> Hi,
>
> I'm doing some research that involves pulling data stored in a mysql clus=
ter
> directly for a map reduce job, without storing the data in HDFS.
>
> I'd like to run hadoop task tracker nodes directly on the mysql cluster
> nodes. The purpose of this being, starting mappers directly in the node
> closest to the data if possible (data locality).
>
> I notice that with HDFS, since the name node knows exactly where each dat=
a
> block is, it uses this to achieve data locality.
>
> Is there a way to achieve my requirement possibly by extending the name n=
ode
> or otherwise?
>
> Thanks in advance.
>
> --
> Regards,
>
> Tharindu
>
> blog: http://mackiemathew.com/
>


--=20
Harsh J