hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lilley <john.lil...@redpoint.net>
Subject RE: HDFS read stats
Date Sun, 09 Feb 2014 14:27:00 GMT
Thanks!  I would have never found that.
john

From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Monday, January 27, 2014 4:57 PM
To: common-user@hadoop.apache.org
Subject: Re: HDFS read stats

FSDataInputStream has this javadoc:
/** Utility that wraps a {@link FSInputStream} in a {@link DataInputStream}

You can utilize this method:
  @InterfaceAudience.LimitedPrivate({"HDFS"})
  public InputStream getWrappedStream() {
    return in;
And cast the return value to DFSInputStream

Cheers

On Mon, Jan 27, 2014 at 11:07 AM, John Lilley <john.lilley@redpoint.net<mailto:john.lilley@redpoint.net>>
wrote:
Ummm... so if I've called FileSystem.open() with an hdfs:// path, and it returns an FSDataInputStream,
how do I get from there to the DFSInputStream that you say has the interface I want?
Thanks
John


From: Ted Yu [mailto:yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>]
Sent: Sunday, January 26, 2014 6:16 PM

To: common-user@hadoop.apache.org<mailto:common-user@hadoop.apache.org>
Subject: Re: HDFS read stats

Please take a look at DFSInputStream#ReadStatistics which contains four metrics including
local bytes read.
You can obtain ReadStatistics through getReadStatistics()

Cheers

On Sun, Jan 26, 2014 at 4:00 PM, John Lilley <john.lilley@redpoint.net<mailto:john.lilley@redpoint.net>>
wrote:
Ted,
Thanks for link!  I says 2.1.0 beta fix, and I can find FileSystem$Statistics class in 2.2.0
but it only seems to talk about read/write ops and bytes, not the local-vs-remote bytes. 
What am I missing?
John


From: Ted Yu [mailto:yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>]
Sent: Sunday, January 26, 2014 10:26 AM
To: common-user@hadoop.apache.org<mailto:common-user@hadoop.apache.org>
Subject: Re: HDFS read stats

Please take a look at https://issues.apache.org/jira/browse/HDFS-4698

Cheers

On Sun, Jan 26, 2014 at 9:11 AM, John Lilley <john.lilley@redpoint.net<mailto:john.lilley@redpoint.net>>
wrote:
Is there a way to monitor the proportion of HDFS read data that is satisfied by local nodes
vs going across the network?
Thanks
John






Mime
View raw message