Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 63605EB8E for ; Mon, 25 Feb 2013 07:41:59 +0000 (UTC) Received: (qmail 91256 invoked by uid 500); 25 Feb 2013 07:41:54 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 90966 invoked by uid 500); 25 Feb 2013 07:41:53 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 90946 invoked by uid 99); 25 Feb 2013 07:41:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Feb 2013 07:41:53 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of psybers@gmail.com designates 209.85.215.54 as permitted sender) Received: from [209.85.215.54] (HELO mail-la0-f54.google.com) (209.85.215.54) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Feb 2013 07:41:48 +0000 Received: by mail-la0-f54.google.com with SMTP id gw10so2382965lab.13 for ; Sun, 24 Feb 2013 23:41:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=fFuScpjThgCR2wUZtUcq2sFIE5IULyaCzs+ODOGmKVk=; b=tuUjlmSHYNChOQINSKeocEPBZ+IZYRt6twA6O3f9v6tIybcO70ryWFhkFF6GGdUHpB ZWPujRRDgZTSlcVk35gFPBFPuxRcefiPAOeqHr+KmJ1N3NSS3P6n0Gabzzi9BDBgqohT B9RItZIqPdIUGAVYAc+CChoGXCLIUz6Z4w5t09Tya9BLpKMXAUVguqM010IVS+XrAtsp fQpQxPHEzkAQKUmG4hqomCvThE255oXusf0ct0vSordhvkkbpaAr0KzQ5Ryo70/14Gzm uaHS7bnhIaC8VIbieRGzac5A90PyQWHv+JqqgBexsXGWotItgUWSjN9A4FgOFPeLg1rr YehA== MIME-Version: 1.0 X-Received: by 10.112.48.163 with SMTP id m3mr3959045lbn.90.1361778086912; Sun, 24 Feb 2013 23:41:26 -0800 (PST) Sender: psybers@gmail.com Received: by 10.112.130.71 with HTTP; Sun, 24 Feb 2013 23:41:26 -0800 (PST) In-Reply-To: References: Date: Mon, 25 Feb 2013 01:41:26 -0600 X-Google-Sender-Auth: bM1TH8TfQsvQID6RlHIgTjD4gug Message-ID: Subject: Re: Slow MR time and high network utilization with all local data From: Robert Dyer To: Harsh J Cc: "" Content-Type: multipart/alternative; boundary=bcaec554d7fe2224b304d687abcd X-Virus-Checked: Checked by ClamAV on apache.org --bcaec554d7fe2224b304d687abcd Content-Type: text/plain; charset=ISO-8859-1 I am using Ganglia. Note I have short circuit reads enabled (I think, I never verified it was working but I do get errors if I run jobs as another user). Also, if Ganglia's network use included the local socket then I would see network utilization in all cases. I see no utilization when using HBase as MR input and MapFile. I also see a small amount when using HBase for both (as one would expect). On Mon, Feb 25, 2013 at 1:22 AM, Harsh J wrote: > Hi Robert, > > How are you measuring the network usage? Note that unless short > circuit reading is on, data reads are done over a local socket as > well, and may appear in network traffic observing tools too (but do > not mean they are over the network). > > On Mon, Feb 25, 2013 at 2:35 AM, Robert Dyer wrote: > > I have a small 6 node dev cluster. I use a 1GB SequenceFile as input to > a > > MapReduce job, using a custom split size of 10MB (to increase the number > of > > maps). Each map call will read random entries out of a shared MapFile > (that > > is around 50GB). > > > > I set replication to 6 on both of these files, so all of the data should > be > > local for each map task. I verified via fsck that no blocks are > > under-replicated. > > > > Despite this, for some reason the MR job maxes out the network and takes > an > > extremely long time. What could be causing this? > > > > Note that the total number of map outputs for this job is around 400 and > the > > reducer just passes the values through, so there shouldn't be much > network > > utilized by the output. > > > > As an experiment, I switched from the SeqFile input to an HBase table and > > now see almost no network used. I also tried leaving the SeqFile as > input > > and switched the MapFile to an HBase table and see about 30% network used > > (which makes sense, as now that 50GB data isn't always local). > > > > What is going on here? How can I debug to see what data is being > > transferred over the network? > > > > -- > Harsh J > -- Robert Dyer rdyer@iastate.edu --bcaec554d7fe2224b304d687abcd Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I am using Ganglia.

Note I have sho= rt circuit reads enabled (I think, I never verified it was working but I do= get errors if I run jobs as another user).

Also, if Ganglia&#= 39;s network use included the local socket then I would see network utiliza= tion in all cases.=A0 I see no utilization when using HBase as MR input and= MapFile.=A0 I also see a small amount when using HBase for both (as one wo= uld expect).


On Mon,= Feb 25, 2013 at 1:22 AM, Harsh J <harsh@cloudera.com> wrot= e:
Hi Robert,

How are you measuring the network usage? Note that unless short
circuit reading is on, data reads are done over a local socket as
well, and may appear in network traffic observing tools too (but do
not mean they are over the network).

On Mon, Feb 25, 2013 at 2:35 AM, Robert Dyer <psybers@gmail.com> wrote:
> I have a small 6 node dev cluster. =A0I use a 1GB SequenceFile as inpu= t to a
> MapReduce job, using a custom split size of 10MB (to increase the numb= er of
> maps). =A0Each map call will read random entries out of a shared MapFi= le (that
> is around 50GB).
>
> I set replication to 6 on both of these files, so all of the data shou= ld be
> local for each map task. =A0I verified via fsck that no blocks are
> under-replicated.
>
> Despite this, for some reason the MR job maxes out the network and tak= es an
> extremely long time. =A0What could be causing this?
>
> Note that the total number of map outputs for this job is around 400 a= nd the
> reducer just passes the values through, so there shouldn't be much= network
> utilized by the output.
>
> As an experiment, I switched from the SeqFile input to an HBase table = and
> now see almost no network used. =A0I also tried leaving the SeqFile as= input
> and switched the MapFile to an HBase table and see about 30% network u= sed
> (which makes sense, as now that 50GB data isn't always local).
>
> What is going on here? =A0How can I debug to see what data is being > transferred over the network?



--
Harsh J



--

Robert Dyer
rdyer@iastate.edu
--bcaec554d7fe2224b304d687abcd--