hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From roman kolcun <roman.w...@gmail.com>
Subject Re: File Chunk to Map Thread Association
Date Thu, 20 Aug 2009 09:59:46 GMT
On Thu, Aug 20, 2009 at 10:30 AM, Harish Mallipeddi <
harish.mallipeddi@gmail.com> wrote:

> On Thu, Aug 20, 2009 at 2:39 PM, roman kolcun <roman.wsmo@gmail.com>
> wrote:
>
> >
> > Hello Harish,
> >
> > I know that TaskTracker creates separate threads (up to
> > mapred.tasktracker.map.tasks.maximum) which execute the map() function.
> > However, I haven't found the piece of code which associate FileSplit with
> > the given map thread. Is it downloaded locally in the TaskTracker
> function
> > or in MapTask?
> >
> >
> >
> Yes this is done by the MapTask.


Thanks, I will have a better look into it.

>
>
> >
> > I know I can increase the input file size by changing
> > 'mapred.min.split.size' , however, the file is split sequentially and
> very
> > rarely two consecutive HDFS blocks are stored on a single node. This
> means
> > that the data locality will not be exploited cause every map() will have
> to
> > download part of the file from network.
> >
> > Roman Kolcun
> >
>
> I see what you mean - you want to modify the hadoop code to allocate
> multiple (non-sequential) data-local blocks to one MapTask.


That's exactly what I want to do.


> I don't know if you'll achieve much by doing all that work.


Basically I would like to emulate larger DFS blocksize. I've performed 2
word count benchmarks on a cluster of 10 machines with 100GB file. With 64MB
blocksize it took 2035 seconds, when I've increased it to 256MB it took 1694
seconds - which is 16.76% increase.


> Hadoop lets you reuse the
> launched JVMs for multiple MapTasks. That should minimize the overhead of
> launching MapTasks.
> Increasing the DFS blocksize for the input files is another means to
> achieve
> the same effect.
>
> Do you think that this could be eliminated by reusing JVMs?
I am doing it as a project for my university degree so I really hope it will
lower the processing time significantly. I would like to make it general for
different block sizes.

Thank you for your help.

Roman Kolcun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message