Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of justin.woody@gmail.com
 designates 209.85.212.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALEj8eOyp88o=fg5ygz7AoE2A0AciXcr1a-PwWt9P1_VXkM6kA@mail.gmail.com>
References: 
 <CALEj8eOyp88o=fg5ygz7AoE2A0AciXcr1a-PwWt9P1_VXkM6kA@mail.gmail.com>
Date: Fri, 14 Oct 2011 11:49:25 -0400
Message-ID: 
 <CABGo67UU4m=x7QNwm5zfsYeQ9bTYWFBGPEJCLWboV2gPNxDGjw@mail.gmail.com>
Subject: Re: Problems Mapping multigigabyte file
From: Justin Woody <justin.woody@gmail.com>
To: mapreduce-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Steve,

Is the input file splittable?

Justin

On Fri, Oct 14, 2011 at 11:23 AM, Steve Lewis <lordjoe2000@gmail.com> wrote=
:
> I have an MR task which runs well with a single input file or an input
> directory with dozens of 50MB input files.
> When the data is in a single input file of 1 GB of more the mapper never
> gets to 0%. There are not errors but when I look at the cluster, the CPUs
> are spending huge amounts of time in a wait state. The job runs when the
> input is 800MB and can complete even=A0with=A0a number of 500MB files as =
input.
> The cluster (0.02) has 8 nodes - 8 cpu per node. Block size is 64MB.
> Any bright ideas
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>