Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 23848D57F for ; Tue, 21 May 2013 19:00:14 +0000 (UTC) Received: (qmail 60097 invoked by uid 500); 21 May 2013 19:00:09 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 60013 invoked by uid 500); 21 May 2013 19:00:09 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 59997 invoked by uid 99); 21 May 2013 19:00:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2013 19:00:09 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ian@cloudera.com designates 209.85.216.47 as permitted sender) Received: from [209.85.216.47] (HELO mail-qa0-f47.google.com) (209.85.216.47) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2013 19:00:04 +0000 Received: by mail-qa0-f47.google.com with SMTP id o13so2373788qaj.20 for ; Tue, 21 May 2013 11:59:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to:x-mailer:x-gm-message-state; bh=OLUrxGHVhVfDfgRk4mE5iRHkMz/Yxp6P+y4gRwoX43I=; b=Ima7/551vFe2aqz6OQEyQU0s0FW1YKRN0fFaSS17krYQEt4bzZIWRJX7jHSsvRJUf0 muL0EReeiN9fnjQhHFKepGcw9MLId0LGeI3Ui78mQlVCvGiTOUqk+bovG0mF/rObpgCH g4WSblo3K/eNMV/s9a/WnQ0HehAeSnXPe/zFtIz94cQhI7dLFTCDKucu9MRxCrVpYFGU Im0U01EL9Z/IN7DLACGy2OEDxLYcyaxjt1a66UrPsPeQNBED4lsRHkhuRMBBgmvZ1L0A MYLeDYuwlLaikg3TOEGpuS0DntWmeu7T4fQ2Uf3l06MUdVgi/wUjP0zdxsccVDjrPyxt MxrQ== X-Received: by 10.224.199.3 with SMTP id eq3mr4059637qab.18.1369162783330; Tue, 21 May 2013 11:59:43 -0700 (PDT) Received: from [192.168.1.4] (c-98-240-93-28.hsd1.tn.comcast.net. [98.240.93.28]) by mx.google.com with ESMTPSA id j10sm3756486qeh.0.2013.05.21.11.59.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 21 May 2013 11:59:42 -0700 (PDT) From: Ian Wrigley Content-Type: multipart/alternative; boundary="Apple-Mail=_25A9311D-87CE-402F-928A-9B0A3E6C8FE0" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Shuffle phase replication factor Date: Tue, 21 May 2013 13:59:41 -0500 References: <869970D71E26D7498BDAC4E1CA92226B6589CDAE@MBX021-E3-NJ-2.exch021.domain.local> To: user@hadoop.apache.org In-Reply-To: <869970D71E26D7498BDAC4E1CA92226B6589CDAE@MBX021-E3-NJ-2.exch021.domain.local> X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQmyvsqxYXTep32bg9w6kkP/cgyYIFMt59s8np4f8SqO55o7F2hWVhp9zzNjC5qCYzQJMq6K X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_25A9311D-87CE-402F-928A-9B0A3E6C8FE0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Intermediate data is written to local disk, not to HDFS. Ian. On May 21, 2013, at 1:57 PM, John Lilley = wrote: > When MapReduce enters =93shuffle=94 to partition the tuples, I am = assuming that it writes intermediate data to HDFS. What replication = factor is used for those temporary files? > john > =20 --- Ian Wrigley Sr. Curriculum Manager Cloudera, Inc Cell: (323) 819 4075 --Apple-Mail=_25A9311D-87CE-402F-928A-9B0A3E6C8FE0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Intermediate data is written to = local disk, not to = HDFS.

Ian.

On May 21, 2013, = at 1:57 PM, John Lilley <john.lilley@redpoint.net> = wrote:

When MapReduce enters =93shuffle=94 to partition = the tuples, I am assuming that it writes intermediate data to = HDFS.  What replication factor is used for those temporary = files?

---
Ian Wrigley
Sr. Curriculum = Manager
Cloudera, Inc
Cell: (323) 819 4075

= --Apple-Mail=_25A9311D-87CE-402F-928A-9B0A3E6C8FE0--