Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CFA8110871 for ; Sat, 21 Sep 2013 13:57:17 +0000 (UTC) Received: (qmail 81003 invoked by uid 500); 21 Sep 2013 13:56:15 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 80832 invoked by uid 500); 21 Sep 2013 13:56:10 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 80711 invoked by uid 99); 21 Sep 2013 13:56:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Sep 2013 13:56:03 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of john.lilley@redpoint.net designates 206.225.164.221 as permitted sender) Received: from [206.225.164.221] (HELO hub021-nj-5.exch021.serverdata.net) (206.225.164.221) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Sep 2013 13:55:57 +0000 Received: from MBX021-E3-NJ-2.exch021.domain.local ([10.240.4.78]) by HUB021-NJ-5.exch021.domain.local ([10.240.4.89]) with mapi id 14.03.0123.003; Sat, 21 Sep 2013 06:55:35 -0700 From: John Lilley To: "user@hadoop.apache.org" Subject: connection overload strategies Thread-Topic: connection overload strategies Thread-Index: Ac620fdbLzDCaz/yQL+klowq8rLYvA== Date: Sat, 21 Sep 2013 13:55:35 +0000 Message-ID: <869970D71E26D7498BDAC4E1CA92226B86D385C4@MBX021-E3-NJ-2.exch021.domain.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [173.160.43.61] Content-Type: multipart/alternative; boundary="_000_869970D71E26D7498BDAC4E1CA92226B86D385C4MBX021E3NJ2exch_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_869970D71E26D7498BDAC4E1CA92226B86D385C4MBX021E3NJ2exch_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable If my YARN application tasks are all reading/writing HDFS simultaneously an= d some node is unable to honor a connection request because it is overloade= d, what happens? I've seen HDFS attempt to retry connections. For that matter, how does MR under YARN deal with connection overload durin= g the shuffle phase, where it seems that such cases would be likely? Thanks John --_000_869970D71E26D7498BDAC4E1CA92226B86D385C4MBX021E3NJ2exch_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

If my YARN application tasks are all reading/writing= HDFS simultaneously and some node is unable to honor a connection request = because it is overloaded, what happens?  I’ve seen HDFS attempt = to retry connections.

For that matter, how does MR under YARN deal with co= nnection overload during the shuffle phase, where it seems that such cases = would be likely?

Thanks

John

 

 

--_000_869970D71E26D7498BDAC4E1CA92226B86D385C4MBX021E3NJ2exch_--