Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0BFE210D18 for ; Mon, 27 Jan 2014 19:07:37 +0000 (UTC) Received: (qmail 82504 invoked by uid 500); 27 Jan 2014 19:07:28 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 82309 invoked by uid 500); 27 Jan 2014 19:07:28 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 82299 invoked by uid 99); 27 Jan 2014 19:07:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jan 2014 19:07:28 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of john.lilley@redpoint.net designates 206.225.164.233 as permitted sender) Received: from [206.225.164.233] (HELO hub021-nj-8.exch021.serverdata.net) (206.225.164.233) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jan 2014 19:07:23 +0000 Received: from MBX021-E3-NJ-2.exch021.domain.local ([10.240.4.78]) by HUB021-NJ-8.exch021.domain.local ([10.240.4.117]) with mapi id 14.03.0158.001; Mon, 27 Jan 2014 11:07:02 -0800 From: John Lilley To: "user@hadoop.apache.org" Subject: RE: HDFS read stats Thread-Topic: HDFS read stats Thread-Index: Ac8auZOCZ8SgawGXTjudSjaIIayagwARS68AAAMMmKAADVgFgAAUlc4w Date: Mon, 27 Jan 2014 19:07:01 +0000 Message-ID: <869970D71E26D7498BDAC4E1CA92226B86E20C49@MBX021-E3-NJ-2.exch021.domain.local> References: <869970D71E26D7498BDAC4E1CA92226B86E1FEA0@MBX021-E3-NJ-2.exch021.domain.local> <869970D71E26D7498BDAC4E1CA92226B86E20144@MBX021-E3-NJ-2.exch021.domain.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [173.160.43.60] Content-Type: multipart/alternative; boundary="_000_869970D71E26D7498BDAC4E1CA92226B86E20C49MBX021E3NJ2exch_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_869970D71E26D7498BDAC4E1CA92226B86E20C49MBX021E3NJ2exch_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Ummm... so if I've called FileSystem.open() with an hdfs:// path, and it re= turns an FSDataInputStream, how do I get from there to the DFSInputStream t= hat you say has the interface I want? Thanks John From: Ted Yu [mailto:yuzhihong@gmail.com] Sent: Sunday, January 26, 2014 6:16 PM To: common-user@hadoop.apache.org Subject: Re: HDFS read stats Please take a look at DFSInputStream#ReadStatistics which contains four met= rics including local bytes read. You can obtain ReadStatistics through getReadStatistics() Cheers On Sun, Jan 26, 2014 at 4:00 PM, John Lilley > wrote: Ted, Thanks for link! I says 2.1.0 beta fix, and I can find FileSystem$Statisti= cs class in 2.2.0 but it only seems to talk about read/write ops and bytes,= not the local-vs-remote bytes. What am I missing? John From: Ted Yu [mailto:yuzhihong@gmail.com] Sent: Sunday, January 26, 2014 10:26 AM To: common-user@hadoop.apache.org Subject: Re: HDFS read stats Please take a look at https://issues.apache.org/jira/browse/HDFS-4698 Cheers On Sun, Jan 26, 2014 at 9:11 AM, John Lilley > wrote: Is there a way to monitor the proportion of HDFS read data that is satisfie= d by local nodes vs going across the network? Thanks John --_000_869970D71E26D7498BDAC4E1CA92226B86E20C49MBX021E3NJ2exch_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Ummm… so if I’= ;ve called FileSystem.open() with an hdfs:// path, and it returns an FSData= InputStream, how do I get from there to the DFSInputStream that you say has the interface I want?

Thanks<= /p>

John

 <= /p>

 <= /p>

From: Ted Yu [= mailto:yuzhihong@gmail.com]
Sent: Sunday, January 26, 2014 6:16 PM
To: common-user@hadoop.apache.org
Subject: Re: HDFS read stats

 

Please take a look at DFSInputStream#ReadStatistics = which contains four metrics including local bytes read.

You can obtain ReadStatistics through getReadStatist= ics()

 

Cheers

 

On Sun, Jan 26, 2014 at 4:00 PM, John Lilley <john.lilley@redp= oint.net> wrote:

Ted,

Thanks for link!  I says 2.1.0 bet= a fix, and I can find FileSystem$Statistics class in 2.2.0 but it only seems to talk about read/write ops and bytes, not the local-vs-rem= ote bytes.  What am I missing?

John

 

 

From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Sunday, January 26, 2014 10:26 AM
To: common-user@hadoop.apache.org
Subject: Re: HDFS read stats

 

Please take a look at https://issues.apache.org/jira/brows= e/HDFS-4698

 

Cheers

 

On Sun, Jan 26, 2014 at 9:11 AM, John Lilley <john.lilley@redpoint.net&g= t; wrote:

Is there a way to monitor the proportion of HDFS read data that is= satisfied by local nodes vs going across the network?

Thanks

John

 

 

 

 

--_000_869970D71E26D7498BDAC4E1CA92226B86E20C49MBX021E3NJ2exch_--