Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 78A4D3B52 for ; Mon, 2 May 2011 07:16:59 +0000 (UTC) Received: (qmail 13276 invoked by uid 500); 2 May 2011 07:16:56 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 13083 invoked by uid 500); 2 May 2011 07:16:56 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 13069 invoked by uid 99); 2 May 2011 07:16:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 07:16:55 +0000 X-ASF-Spam-Status: No, hits=3.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tmatthewjohn1988@gmail.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 07:16:49 +0000 Received: by vws7 with SMTP id 7so6340470vws.35 for ; Mon, 02 May 2011 00:16:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=4FyCP+mJ1Jk4m5iSH2q4w1XcEIcAlyvrOib7m5B0LRU=; b=Lc+Uw4wwIgSG88U+Dgl5UI8aGLiaihpWfVZ0cBfiq1q2ANWdwd5k1I/Lg1K2HKbW2C iE0uAwouYHnMwiXg/INgCwoloBGnr9WAya/SewWfoj59yYOz7ryH6ElgqOO4ONcAzFZb ywMakqyG+n2dIF7mHIxl9R/EPtp0Fk+WbSW1w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=u9g16ptJPQ0tgHQT8qwyUxhyaQvSxyocBkJmP8dgvgqqkJWW/pqiEX8zThphoN/VOh jIjDzRWQjenbFswUwycGf+4SsZAC/19pwb6EwbFeGZEpcC5eEjAnAsOV7Z0MZAcGbDIq tDshwgmmafEHrgMEQs1T+WENma/24gVOsKex4= MIME-Version: 1.0 Received: by 10.52.97.166 with SMTP id eb6mr15009vdb.185.1304320589007; Mon, 02 May 2011 00:16:29 -0700 (PDT) Received: by 10.52.165.226 with HTTP; Mon, 2 May 2011 00:16:28 -0700 (PDT) In-Reply-To: References: Date: Mon, 2 May 2011 12:46:28 +0530 Message-ID: Subject: Re: HDFS - MapReduce coupling From: Matthew John To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf307cfcb6618aa304a245cdae --20cf307cfcb6618aa304a245cdae Content-Type: text/plain; charset=ISO-8859-1 Any documentations on how the different daemons do the write/read on HDFS and Local File System (direct), I mean the different protocols used in the interactions. I basically wanted to figure out how intricate the coupling between the Storage (HDFS + Local) and other processes in the Hadoop infrastructure is. On Mon, May 2, 2011 at 12:26 PM, Ted Dunning wrote: > Yes. There is quite a bit of need for the local file system in clustered > mode. > > For one think, all of the shuffle intermediate files are on local disk. > For > another, the distributed cache is actually stored on local disk. > > HFDS is a frail vessel that cannot cope with all the needs. > > On Sun, May 1, 2011 at 11:48 PM, Matthew John >wrote: > > > ... > > 2) Does the Hadoop system utilize the local storage directly for any > > purpose > > (without going through the HDFS) in clustered mode? > > > > > --20cf307cfcb6618aa304a245cdae--