Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AEABA19ACD for ; Tue, 26 Apr 2016 20:31:27 +0000 (UTC) Received: (qmail 63693 invoked by uid 500); 26 Apr 2016 20:31:25 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 63576 invoked by uid 500); 26 Apr 2016 20:31:25 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 63550 invoked by uid 99); 26 Apr 2016 20:31:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Apr 2016 20:31:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0C62AC2068 for ; Tue, 26 Apr 2016 20:31:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.821 X-Spam-Level: X-Spam-Status: No, score=-0.821 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id xXqP5q_Bibdg for ; Tue, 26 Apr 2016 20:31:22 +0000 (UTC) Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 65B105F24A for ; Tue, 26 Apr 2016 20:31:22 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id zm5so11854802pac.0 for ; Tue, 26 Apr 2016 13:31:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=of2SVTlFtDfLastav+u+IDJjH3izAo1Scy4MQ5oBMFs=; b=wofYxEqHfbpcVGw+xzGAWqxw/eHJ3irdnt03yxf72fmfwzBIx3+0za2XQsiKLxQs4l AGA0syqn3e50uUAWqpwGJqjlxtYGxt8mQ4tRW+tuRSrI+AK9q3VK6dQ1mqu4aeEYNmiY XV7X0Rzsl4nL3z2ZApIiWxlfXd/AjfkghCQjHxKGkPGcyXWJ1UdZ6ihausYv0awKpHso UUCn7sThjX75/KMxel7HQ3/WIcB9ydz0Nr+x76Ye620MQq2EoSJni6KIFLrdA92dJ7Rq BDJoIu2lcmPzqk2nan81gJcKX3hIE08bLHZKF4Mf0hq2R+HZohh1EPj6rzuEk3O93/CC O3Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=of2SVTlFtDfLastav+u+IDJjH3izAo1Scy4MQ5oBMFs=; b=XHBp3q70H3Y0BNfIUERtXqgOhSSg158FPB7NHjnVoU7mfmfbzR6W20ZJDat/lGDqzV Ao4am2UO2UppaRNdE3BiR7cHLegSOgmbOwTyKqwleK+XJ/slmlbqVkfzqW0BAJ5SqL25 8ytibxvYdATZWGpp6yfojAbnCOwaJN63pVcdEN4HzKtXhEnFGpE7yXXwTzWa2RWPx20C +8gfIJzhMuQ9uwyS5myIux4nGYaN44YNzlTn9mNbopUcgqruFZZ7jCKp+fBP8tZlsOJ9 LwV23iv9hyJMbf+QyS3g3jkpKaY8B+4PVxJIENKmWx3DQeEhS2Y6yLucRWoCt6HdFClF cp0g== X-Gm-Message-State: AOPr4FX32lzMbF8GWermVjgs1vVMUCezy5TGaN+vniQmFsjMHPiilndS/GXqU5xrFRshSg== X-Received: by 10.66.124.226 with SMTP id ml2mr6456097pab.90.1461702675401; Tue, 26 Apr 2016 13:31:15 -0700 (PDT) Received: from [21.26.52.7] ([172.56.39.236]) by smtp.gmail.com with ESMTPSA id fn3sm637429pab.20.2016.04.26.13.31.13 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 26 Apr 2016 13:31:13 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: HDFS as Shuffle Service From: Timothy Chen X-Mailer: iPhone Mail (13E238) In-Reply-To: Date: Tue, 26 Apr 2016 13:31:12 -0700 Cc: dev@spark.apache.org Content-Transfer-Encoding: quoted-printable Message-Id: <1511E1C3-0D95-4022-86BE-07C045AFB0C4@gmail.com> References: To: Michael Gummelt Are you suggesting to have shuffle service persist and fetch data with hdfs,= or skip shuffle service altogether and just write to hdfs? Tim > On Apr 26, 2016, at 11:20 AM, Michael Gummelt wro= te: >=20 > Has there been any thought or work on this (or any other networked file sy= stem)? It would be valuable to support dynamic allocation without depending= on the shuffle service. >=20 > --=20 > Michael Gummelt > Software Engineer > Mesosphere --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org For additional commands, e-mail: dev-help@spark.apache.org