Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3569810E08 for ; Sat, 15 Jun 2013 13:40:31 +0000 (UTC) Received: (qmail 29154 invoked by uid 500); 15 Jun 2013 13:40:25 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 28846 invoked by uid 500); 15 Jun 2013 13:40:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 28839 invoked by uid 99); 15 Jun 2013 13:40:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Jun 2013 13:40:18 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of john.lilley@redpoint.net designates 206.225.164.221 as permitted sender) Received: from [206.225.164.221] (HELO hub021-nj-5.exch021.serverdata.net) (206.225.164.221) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Jun 2013 13:40:12 +0000 Received: from MBX021-E3-NJ-2.exch021.domain.local ([10.240.4.78]) by HUB021-NJ-5.exch021.domain.local ([10.240.4.89]) with mapi id 14.03.0123.003; Sat, 15 Jun 2013 06:39:51 -0700 From: John Lilley To: "user@hadoop.apache.org" Subject: RE: Shuffle design: optimization tradeoffs Thread-Topic: Shuffle design: optimization tradeoffs Thread-Index: Ac5mvK4U3AqXySAuSy2q9XcbmZvN+QAaRCOAABdmrMAAGKJdgAB4x8Hg Date: Sat, 15 Jun 2013 13:39:51 +0000 Message-ID: <869970D71E26D7498BDAC4E1CA92226B658C64C1@MBX021-E3-NJ-2.exch021.domain.local> References: <869970D71E26D7498BDAC4E1CA92226B658C2BE4@MBX021-E3-NJ-2.exch021.domain.local> <1370986305.13124.131.camel@auk59.llnl.gov> <869970D71E26D7498BDAC4E1CA92226B658C4086@MBX021-E3-NJ-2.exch021.domain.local> <1371068829.13124.155.camel@auk59.llnl.gov> In-Reply-To: <1371068829.13124.155.camel@auk59.llnl.gov> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [173.160.43.61] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org QWxiZXJ0LA0KVGhhbmtzIGZvciB0aGUgbGluay4gIFRoaXMgaXMgaW5kZWVkIHdoYXQgSSBhbSB0 YWxraW5nIGFib3V0Lg0KVGhlIGF1dGhvcnMgaGF2ZSB0YWtlbiB0aGUgaWRlYSBldmVuIGZ1cnRo ZXIsIGF2b2lkaW5nIGRpc2sgd3JpdGVzIG9uIGVpdGhlciB0aGUgbWFwcGVyIG9yIHJlZHVjZXIg c2lkZS4gIEl0J3Mgbm90IGNsZWFyIHRvIG1lIHRoYXQgdGhpcyBzY2FsZXMgd2VsbCB0byAxMDAw cyBvZiBub2RlcyBob3dldmVyLCBhcyB0aGUgZG93bnNpZGUgdG8gbm90IGxhbmRpbmcgZGF0YSBv biBkaXNrIG9uIHRoZSByZWR1Y2VyIHNpZGUgaXMgdGhhdCBpdCB3b3VsZCBzZWVtIHRvIGltcG9z ZSBhdCBsZWFzdCBvbmUgb2YgdGhlIGZvbGxvd2luZyByZXF1aXJlbWVudHM6DQotLSBhIGxvdCBv ZiBtZW1vcnkgb24gdGhlIHJlZHVjZXIgc2lkZQ0KLS0gcmVkdWNlcnMga2VlcCBjb25uZWN0aW9u cyBvcGVuIHRvIHJldHJpZXZlIG1hcCBmaWxlIGRhdGENCi0tIHJlZHVjZXIvbWFwLWZpbGUgY29u bmVjdGlvbnMgYXJlIGp1Z2dsZWQgc28gYXMgdG8gYXZvaWQga2VlcGluZyB0b28gbWFueSBvcGVu IGF0IG9uY2UuDQpKb2huDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBBbGJl cnQgQ2h1IFttYWlsdG86Y2h1MTFAbGxubC5nb3ZdIA0KU2VudDogV2VkbmVzZGF5LCBKdW5lIDEy LCAyMDEzIDI6MjcgUE0NClRvOiB1c2VyQGhhZG9vcC5hcGFjaGUub3JnDQpTdWJqZWN0OiBSRTog U2h1ZmZsZSBkZXNpZ246IG9wdGltaXphdGlvbiB0cmFkZW9mZnMNCg0KT24gV2VkLCAyMDEzLTA2 LTEyIGF0IDE4OjA4ICswMDAwLCBKb2huIExpbGxleSB3cm90ZToNCj4gSW4gcmVhZGluZyB0aGlz IGxpbmsgYXMgd2VsbCBhcyB0aGUgc2FpbGZpc2ggcmVwb3J0LCBpdCBzdHJpa2VzIG1lIA0KPiB0 aGF0IEhhZG9vcCBza2lwcGVkIGEgcG90ZW50aWFsbHkgc2lnbmlmaWNhbnQgb3B0aW1pemF0aW9u LiAgTmFtZWx5LCANCj4gd2h5IGFyZSBtdWx0aXBsZSBzb3J0ZWQgc3BpbGwgZmlsZXMgbWVyZ2Vk IGludG8gYSBzaW5nbGUgb3V0cHV0IGZpbGU/DQo+IFdoeSBub3QgaGF2ZSB0aGUgYXV4aWxpYXJ5 IHNlcnZpY2UgbWVyZ2Ugb24gdGhlIGZseSwgdGh1cyBhdm9pZGluZyANCj4gbGFuZGluZyB0aGVt IHRvIGRpc2s/DQoNCkkgYmVsaWV2ZSB3aGF0IHlvdSdyZSB0YWxraW5nIGFib3V0L3N1Z2dlc3Rp bmcgaXMgc2ltaWxhciB0byB3aGF0J3MgZGlzY3Vzc2VkIGluIHRoaXMgcGFwZXI/DQoNCmh0dHA6 Ly9wYXNsLmVuZy5hdWJ1cm4uZWR1L3B1YnMvc2MxMS1uZXRsZXYucGRmDQoNCkFsDQoNCj4gV2Fz IHRoaXMgY29uc2lkZXJlZCBhbmQgcmVqZWN0ZWQgZHVlIHRvIHBsYWNpbmcgbWVtb3J5L0NQVSAN Cj4gcmVxdWlyZW1lbnRzIG9uIHRoZSBhdXhpbGlhcnkgc2VydmljZT8gIEkgYW0gYXNzdW1pbmcg dGhhdCB3aGV0aGVyIHRoZSANCj4gbWVyZ2Ugd2FzIGRvbmUgb24gZGlzayBvciBpbiBhIHN0cmVh bSwgaXQgd291bGQgcmVxdWlyZSANCj4gZGVjb21wcmVzc2lvbi9yZWNvbXByZXNzaW9uIG9mIHRo ZSBkYXRhLg0KPiBKb2huDQo+IA0KPiANCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4g RnJvbTogQWxiZXJ0IENodSBbbWFpbHRvOmNodTExQGxsbmwuZ292XQ0KPiBTZW50OiBUdWVzZGF5 LCBKdW5lIDExLCAyMDEzIDM6MzIgUE0NCj4gVG86IHVzZXJAaGFkb29wLmFwYWNoZS5vcmcNCj4g U3ViamVjdDogUmU6IFNodWZmbGUgZGVzaWduOiBvcHRpbWl6YXRpb24gdHJhZGVvZmZzDQo+IA0K PiBPbiBUdWUsIDIwMTMtMDYtMTEgYXQgMTY6MDAgKzAwMDAsIEpvaG4gTGlsbGV5IHdyb3RlOg0K PiA+IEkgYW0gY3VyaW91cyBhYm91dCB0aGUgdHJhZGVvZmZzIHRoYXQgZHJvdmUgZGVzaWduIG9m IHRoZSANCj4gPiBwYXJ0aXRpb24vc29ydC9zaHVmZmxlIChFbGVwaGFudCBib29rIHAgMjA4KS4g IERvdWJ0bGVzcyB0aGlzIGhhcyANCj4gPiBiZWVuIHR1bmVkIGFuZCBtZWFzdXJlZCBhbmQgcmV0 dW5lZCwgYnV0IEnigJlkIGxpa2UgdG8ga25vdyB3aGF0IA0KPiA+IG9ic2VydmF0aW9ucyBjYW1l IGFib3V0IGR1cmluZyB0aGUgaXRlcmF0aXZlIG9wdGltaXphdGlvbiBwcm9jZXNzIHRvIA0KPiA+ IGRyaXZlIHRoZSBmaW5hbCBkZXNpZ24uICBGb3IgZXhhbXBsZToNCj4gPiANCj4gPiDCtyAgICAg ICAgV2h5IGRvZXMgdGhlIG1hcHBlciBvdXRwdXQgY3JlYXRlIGEgc2luZ2xlIG9yZGVyZWQgZmls ZQ0KPiA+IGNvbnRhaW5pbmcgYWxsIHBhcnRpdGlvbnMsIGFzIG9wcG9zZWQgdG8gYSBmaWxlIHBl ciBncm91cCBvZiANCj4gPiBwYXJ0aXRpb25zICh3aGljaCB3b3VsZCBzZWVtIHRvIGxlbmQgaXRz ZWxmIGJldHRlciB0byBtdWx0aS1jb3JlIA0KPiA+IHNjYWxpbmcpLCBvciBldmVuIGEgZmlsZSBw ZXIgcGFydGl0aW9uPw0KPiANCj4gSSByZXNlYXJjaGVkIHRoaXMgYXdoaWxlIGJhY2sgd29uZGVy aW5nIHRoZSBzYW1lIHRoaW5nLCBhbmQgZm91bmQgdGhpcyANCj4gSklSQQ0KPiANCj4gaHR0cHM6 Ly9pc3N1ZXMuYXBhY2hlLm9yZy9qaXJhL2Jyb3dzZS9IQURPT1AtMzMxDQo+IA0KPiBBbA0KPiAN Cj4gPiDCtyAgICAgICAgV2h5IGRvZXMgdGhlIG1heCBudW1iZXIgb2Ygc3RyZWFtcyB0byBtZXJn ZSBhdCBvbmNlDQo+ID4gKGlzLnNvcnQuZmFjdG9yKSBkZWZhdWx0IHRvIDEwPyAgSXMgdGhpcyBv YnNvbGV0ZT8gIEluIG15IA0KPiA+IGV4cGVyaWVuY2UsIHNvIGxvbmcgYXMgeW91IGhhdmUgbWVt b3J5IHRvIGJ1ZmZlciBlYWNoIGlucHV0IGF0IDFNQiANCj4gPiBvciBzbywgdGhlIG1lcmdlciBp cyBtb3JlIGVmZmljaWVudCBhcyBhIHNpbmdsZSBwaGFzZS4NCj4gPiANCj4gPiDCtyAgICAgICAg V2h5IGRvZXMgdGhlIG1hcHBlciBkbyBhIGZpbmFsIG1lcmdlIG9mIHRoZSBzcGlsbCBmaWxlcyBk bw0KPiA+IGRpc2ssIGluc3RlYWQgb2YgaGF2aW5nIHRoZSBhdXhpbGlhcnkgcHJvY2VzcyAoaW4g WUFSTikgbWVyZ2UgYW5kIA0KPiA+IHN0cmVhbSBkYXRhIG9uIHRoZSBmbHk/DQo+ID4gDQo+ID4g wrcgICAgICAgIFdoeSBkbyBtYXBwZXJzIHNvcnQgdGhlIHR1cGxlcywgYXMgb3Bwb3NlZCB0byBv bmx5DQo+ID4gcGFydGl0aW9uaW5nIHRoZW0gYW5kIGxldHRpbmcgdGhlIHJlZHVjZXJzIGRvIHRo ZSBzb3J0aW5nPw0KPiA+IA0KPiA+IFNvcnJ5IGlmIHRoaXMgaXMgb3Zlcmx5IGFjYWRlbWljLCBi dXQgSeKAmW0gc3VyZSBhIGxvdCBvZiBwZW9wbGUgcHV0IGEgDQo+ID4gbG90IG9mIHRpbWUgaW50 byB0aGUgdHVuaW5nIGVmZm9ydCwgYW5kIEkgaG9wZSB0aGV5IGxlZnQgYSByZWNvcmQgb2YgDQo+ ID4gdGhlaXIgZWZmb3J0cy4NCj4gPiANCj4gPiBUaGFua3MNCj4gPiANCj4gPiBKb2huDQo+ID4g DQo+ID4gIA0KPiA+IA0KPiA+IA0KPiAtLQ0KPiBBbGJlcnQgQ2h1DQo+IGNodTExQGxsbmwuZ292 DQo+IENvbXB1dGVyIFNjaWVudGlzdA0KPiBIaWdoIFBlcmZvcm1hbmNlIFN5c3RlbXMgRGl2aXNp b24NCj4gTGF3cmVuY2UgTGl2ZXJtb3JlIE5hdGlvbmFsIExhYm9yYXRvcnkNCj4gDQo+IA0KLS0N CkFsYmVydCBDaHUNCmNodTExQGxsbmwuZ292DQpDb21wdXRlciBTY2llbnRpc3QNCkhpZ2ggUGVy Zm9ybWFuY2UgU3lzdGVtcyBEaXZpc2lvbg0KTGF3cmVuY2UgTGl2ZXJtb3JlIE5hdGlvbmFsIExh Ym9yYXRvcnkNCg0KDQo=