Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of shahilvarshney@gmail.com
 designates 209.85.212.181 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <D1495D00.1EDA9%cnauroth@hortonworks.com>
References: 
 <CANb=h_gwYO=8MPRP7JsJ+Qdfn-0AXNkRV7NT92ZDviXMFDyABA@mail.gmail.com>
	<CANb=h_ht56ZBnM9i4uuS=4gF1LsbNP7kqpDujq5ENWi510e1yg@mail.gmail.com>
	<D1495D00.1EDA9%cnauroth@hortonworks.com>
Date: Wed, 8 Apr 2015 18:44:26 +0530
Message-ID: 
 <CANb=h_gGvjpouEDYS0=fYZhSk=PsHRU6qjzq9Q3C35kvJd4u6Q@mail.gmail.com>
Subject: Re: map() function call related
From: Shahil Varshney <shahilvarshney@gmail.com>
To: hdfs-dev@hadoop.apache.org, cnauroth@hortonworks.com
Cc: "mapreduce-dev-owner@hadoop.apache.org"
 <mapreduce-dev-owner@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=f46d04426b648569950513364fd8

--f46d04426b648569950513364fd8
Content-Type: text/plain; charset=UTF-8

Sir ,
Actually i want to want to perform de duplication on input splits . so for
this, i have to perform content based chunking (using TTTD algorithm) on
each input split and and leave those chunks that are similar with previous
chunk and send only new chunks to map.

sir please tell me .. in which class should i have to make changes.

On Tue, Apr 7, 2015 at 10:43 PM, Chris Nauroth <cnauroth@hortonworks.com>
wrote:

> Hello Shahil,
>
> In the current trunk codebase, the relevant files are
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-co
> re/src/main/java/org/apache/hadoop/mapred/MapTask.java and
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-co
> re/src/main/java/org/apache/hadoop/mapreduce/Mapper.java.  MapTask manages
> the execution of the mapper task, and eventually it calls Mapper#run,
> which then calls into the implementation of the map method.  BTW, you'll
> also see a corresponding ReduceTask.java and Reducer.java in the same
> directories if you need to look at those too.
>
> Input split calculation is performed by a subclass of InputFormat.
>
> http://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/mapreduce/InputF
> ormat.html
>
>
> I recommend looking at that.  You also can navigate down through those
> JavaDocs to identify subclasses of InputFormat, like FileInputFormat and
> TextInputFormat, which you can then find in the source code.
>
> I hope this helps.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
>
>
>
> On 4/7/15, 6:09 AM, "Shahil Varshney" <shahilvarshney@gmail.com> wrote:
>
> >Sir ,
> >i want to know that which class in hadoop (internal source class) is
> >responsible for calling map function for each key value pair(means calls
> >map() function).
> >
> > and which class actually done the input split job. i want to create my
> >own
> >class for input split so please tell me .
>
>

--f46d04426b648569950513364fd8--