Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D17ACD04C for ; Tue, 11 Dec 2012 04:04:16 +0000 (UTC) Received: (qmail 41788 invoked by uid 500); 11 Dec 2012 04:04:12 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 41521 invoked by uid 500); 11 Dec 2012 04:04:10 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 41464 invoked by uid 99); 11 Dec 2012 04:04:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Dec 2012 04:04:09 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hemanty@thoughtworks.com designates 64.18.0.188 as permitted sender) Received: from [64.18.0.188] (HELO exprod5og109.obsmtp.com) (64.18.0.188) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Dec 2012 04:04:00 +0000 Received: from mail-vc0-f198.google.com ([209.85.220.198]) (using TLSv1) by exprod5ob109.postini.com ([64.18.4.12]) with SMTP ID DSNKUMawmufPYA5u3YkQn/4GcchWRSq4JBXx@postini.com; Mon, 10 Dec 2012 20:03:39 PST Received: by mail-vc0-f198.google.com with SMTP id n11so6221449vch.5 for ; Mon, 10 Dec 2012 20:03:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=hBwDkRbJiJqPSRiEwc6o1c7vACYYhOkUmrc38cgq87U=; b=Sin9hGWf997r47C1/s2v0syBD6OlvxzotmPwHJZthbBBhCSsedhEEkUTCKItDMesL5 bnEnSkYXTsgy2lU4XFnjicw6gkcF4qVIEIdHVg7ElATtvinFFfstKEFhnPv+avirztB0 Xw5RVTTms9Rt6p6JMNVykVaQi0N0SXC4PgLXvYIIX9lweLUyQpx2IBh3SaQ5WLi+7HmL h+ICPzWp4qjkFGmGITKbWVpjChcPchHH1Wyj/mia4nBs6W87pGq6OFXybY0vYrsN8YK6 i9LMtdplyM2zBjJzTbouZSUXpMuLwWjzyYjLPvxgUnYBh6n2uAbL8US9WF9ssnie9WNE qCEA== Received: by 10.58.221.130 with SMTP id qe2mr10717421vec.14.1355198617847; Mon, 10 Dec 2012 20:03:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.58.221.130 with SMTP id qe2mr10717416vec.14.1355198617693; Mon, 10 Dec 2012 20:03:37 -0800 (PST) Received: by 10.58.134.13 with HTTP; Mon, 10 Dec 2012 20:03:37 -0800 (PST) In-Reply-To: <244C4BD5-E0C4-421C-8B19-9D32CB4CDB85@apache.org> References: <343176F5-0EF6-4C64-A4EB-7C0B5ADF4670@gmail.com> <1DD6A29F-062E-4E40-BAF4-964BFDDC2E71@gmail.com> <244C4BD5-E0C4-421C-8B19-9D32CB4CDB85@apache.org> Date: Tue, 11 Dec 2012 09:33:37 +0530 Message-ID: Subject: Re: "attempt*" directories in user logs From: Hemanth Yamijala To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7bf0cabe352f2e04d08bc4ea X-Gm-Message-State: ALoCoQkw5QwIFwp1g42yOlAal9lzJA5MRyRcaI2CvdOv5q37LfIf3Id+kG3yPy8Ms6n/VwPJmYulgFsKW3+c3S495nOOWkimAEFRrE5BmMhRUDwNxRzNhlJMChWL7Adux0FbB1ap2TauaQJ/0ao18GDd6mOBk9/3ag== X-Virus-Checked: Checked by ClamAV on apache.org --047d7bf0cabe352f2e04d08bc4ea Content-Type: text/plain; charset=ISO-8859-1 However, in the case Oleg is talking about the attempts are: attempt_201212051224_0021_m_000000_0 attempt_201212051224_0021_m_000002_0 attempt_201212051224_0021_m_000003_0 These aren't multiple attempts of a single task, are they ? They are actually different tasks. If they were multiple attempts, I would expect the last digit to get incremented, like attempt_201212051224_0021_m_000000_0 and attempt_201212051224_0021_m_000000_1, for instance. It looks like at least 3 different tasks were launched on this node. One of them could be setup task. Oleg, how many map tasks does the Jobtracker UI show for this job. Thanks hemanth On Tue, Dec 11, 2012 at 12:19 AM, Vinod Kumar Vavilapalli < vinodkv@hortonworks.com> wrote: > > MR launches multiple attempts for single Task in case of TaskAttempt > failures or when speculative execution is turned on. In either case, a > given Task will only ever have one successful TaskAttempt whose output will > be accepted (committed). > > Number of reduces is set to 1 by default in mapred-default.xml - you > should explicitly set it to zero if you don't want reducers. > > By master, I suppose you mean JobTracker. JobTracker doesn't show all the > attempts for a given Task, you should navigate to per-task page to see that. > > > Thanks, > +Vinod Kumar Vavilapalli > Hortonworks Inc. > http://hortonworks.com/ > > On Dec 9, 2012, at 6:53 AM, Oleg Zhurakousky wrote: > > I studying user logs on the two node cluster that I have setup and I was > wondering if anyone can shed some light on these "attempt*' directories > > $ ls > > attempt_201212051224_0021_m_000000_0 attempt_201212051224_0021_m_000003_0 > job-acls.xml > attempt_201212051224_0021_m_000002_0 attempt_201212051224_0021_r_000000_0 > > I mean its obvious that its talking about 3 attempts for Map task and 1 > attempt for reduce task. However my current MR job only results in some > output written to "attempt_201212051224_0021_m_000000_0". Nothing is the > reduce part (understandably since I don't even have a reducer, so my > question is: > > 1. The two more M attempts. . . what are they? > 2. Why was there an attempt to do a Reduce when no reducer was > provided.implemented > 3. Why my master node only had 1 attempt for M task but the slave had all > that's displayed and questioned above (the 'ls' output above is from the > slave node) > > Thanks > Oleg > > > --047d7bf0cabe352f2e04d08bc4ea Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable However, in the case Oleg is talking about the attempts are:
attempt_201212051224_0021= _m_00000= 0_0 =A0
attempt_20= 1212051224_0021_m_000002_0
attempt_201212051224_0021_m_000003_0

These aren't multiple attempts of a single task, are they ? They are= actually different tasks. If they were multiple attempts, I would expect t= he last digit to get incremented, like=A0attempt_201212051224_0021_m_000000_0 and=A0= attempt_2012120= 51224_0021_m_000000_1, for instance.

It = looks like at least 3 different tasks were launched on this node. One of th= em could be setup task. Oleg, how many map tasks does the Jobtracker UI sho= w for this job.

Tha= nks
hemanth


On Tue, Dec 1= 1, 2012 at 12:19 AM, Vinod Kumar Vavilapalli <vinodkv@hortonworks.c= om> wrote:
MR launches multiple attempts for single Task in case of TaskAt= tempt failures or when speculative execution is turned on. In either case, = a given Task will only ever have one successful TaskAttempt whose output wi= ll be accepted (committed).

Number of reduces is set to 1 by default in mapred-defa= ult.xml - you should explicitly set it to zero if you don't want reduce= rs.

By master, I suppose you mean JobTracker. JobT= racker doesn't show all the attempts for a given Task, you should navig= ate to per-task page to see that.


Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.<= br>http://hortonworks= .com/

On Dec 9, 2012, at 6:53 AM, Oleg Zhurakousky wrote:

=
I studying user logs on the two node cluster= that I have setup and I was wondering if anyone can shed some light on the= se "attempt*' directories
$ ls
attempt_201212051224_0021_m_= 000000_0 =A0attempt_201212051224_0021_m_000003_0 =A0job-acls.xml
attempt= _201212051224_0021_m_000002_0 =A0attempt_201212051224_0021_r_000000_0
I mean its obvious that its talking about 3 attempts for Map task and 1 a= ttempt for reduce task. However my current MR job only results in some outp= ut written to "attempt_201212051224_0021_m_000000_0". Nothing is = the reduce part (understandably since I don't even have a reducer, so m= y question is:

1. The two more M attempts. . . what are they?
2. Why was there an a= ttempt to do a Reduce when no reducer was provided.implemented
3. Why my= master node only had 1 attempt for M task but the slave had all that's= displayed and questioned above (the 'ls' output above is from the = slave node)

Thanks
Oleg


--047d7bf0cabe352f2e04d08bc4ea--