hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Order of files in Map class
Date Wed, 20 Jul 2011 07:44:11 GMT

Your second example is how it happens in Hadoop, but there's more here
to understand.

To start with, your InputFormat (input splitter) computes and
publishes a set amount of InputSplits. The total number of input
splits is gonna be your total number of 'Map Tasks' in Hadoop as the
job proceeds. The input splits are generally block splits, i.e.,
start-and-stop lengths over the same file.

Each 'MapTask' is designated one split from this list of splits. So
every map task would initialize separately, in its own JVM (no shared
resources -- again, its a different instance of mappers per file or
block!) and read the input split alone, into its map(key, value,
context) function.

So to summarize, your second example is what will happen, but it would
be in parallel instead, such as:

map1 | map2 | …
file1 | file2 | …
row1 | row1 | …
row2 | row 2 | …

P.s. What I've explained here is the default behavior. Of course
things can be highly tweaked to achieve other things, like your first
example, but those probably come with greater read costs attached. The
'hadoop' way is data local, and one-file-per-task.

On Wed, Jul 20, 2011 at 12:11 PM, Florin P <florinpico@yahoo.com> wrote:
> Hello!
>  Suppose that we have the files F1, F2,..Fk given by the input splitter to the map class,
what is the order in which they will arrive when map function  is applied?
>  What is interesting me  if  it is possible that in the map function to arrive mixed
key-value pairs from different files? They keys will arrive related with their file, till
no more keys are left from source file or they can arrive one key from F1 one key from Fk
and so on.
>  Example:
>   Mixed key value pairs at the map function:
>    K1 from F1
>    K5 from F5
>    K7 from F8
>  etc
>  ordered key-value pairs:
>    K1 from F1
>   ..
>    K_end_F1 from F1
>    K5 from F5
> ..
>  K_end_F5 from F5
>  and so on.
> I'll look forward for your answer.
>  Regards,
>  Florin

Harsh J

View raw message