Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPPeW7UaTC0JXz+8thrhE9SbV-HCC9qVvrXY3jNPKEUAyLwyww@mail.gmail.com>
References: 
 <CAPPeW7UaTC0JXz+8thrhE9SbV-HCC9qVvrXY3jNPKEUAyLwyww@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Wed, 5 Dec 2012 23:03:09 +0530
Message-ID: 
 <CAOcnVr0-dvOiG1kna1dH_zoK2dd+5EXgyeTH_8=ya9mDMeXjQQ@mail.gmail.com>
Subject: Re: M/R, Strange behavior with multiple Gzip files
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Cc: gpolaert@cyres.fr
Content-Type: text/plain; charset=ISO-8859-1

Your problem isn't clear in your description - can you please
rephrase/redefine in terms of what you are expecting vs. what you are
observing.

Also note that Gzip files are not splittable by nature of their codec
algorithm, and hence a TextInputFormat over plain/regular Gzip files
would end up spawning and/or processing one whole Gzip file via one
mapper, instead of multiple mappers per file.

On Wed, Dec 5, 2012 at 9:32 PM, x6i4uybz labs <x6i4uyzbz.labs@gmail.com> wrote:
> Hi everybody,
>
> I have a M/R job which does a bulk import to hbase.
> I have to process many gzip files (2800 x ~ 100mb)
>
> I don't understand why my job instanciates 80 maps but runs each map
> sequentialy like if there is only one big gz file.
>
> Is there a problem in my driver ? Or maybe I miss something.
> I use "FileInputFormat.addInputPath(job, new Path(args[0]))" where args[0]
> is a directory.
>
> Can you help me, please ?
>
> Thanks, Guillaume


-- 
Harsh J