Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of bikas@hortonworks.com
 designates 209.85.219.53 as permitted sender)
From: Bikas Saha <bikas@hortonworks.com>
References: 
 <CA+qbEUO8JkTugwkND9zzTyfB338PW2oJZ9XfLFW98pkjKGXCTg@mail.gmail.com>
In-Reply-To: 
 <CA+qbEUO8JkTugwkND9zzTyfB338PW2oJZ9XfLFW98pkjKGXCTg@mail.gmail.com>
MIME-Version: 1.0
Thread-Index: AQGu4Uy1la7aFCUEKpF5Z8jlJObYCJjLDXNw
Date: Tue, 26 Feb 2013 13:47:09 -0800
Message-ID: <a9dc643bada06711faa0b3eedfb45f3e@mail.gmail.com>
Subject: RE: VOTE: HDFS-347 merge
To: hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

In my opinion, this feature of short circuit reads (HDFS-347 or HDFS-2246)
is not a desirable feature for HDFS. We should be working towards removing
this feature instead of enhancing it and making it popular.

Maybe short-circuit reads were something that HBase needed for performance
at a point in time when HDFS performance was slow. But after all the
improvements that have been made, is it still unacceptably slow to read
from HDFS? Is there more good engineering that we can do to close that
gap? Close it for all HDFS users and not just the ones who use
short-circuit reads?
Which brings me to the question - Who is the target audience for this
feature? From what I see, anyone who potentially wants to use it ==
everyone. Now if everyone starts using short circuit reads what happens to
the performance problem that we are trying to solve? Will performance
still be better then? This is especially important in the context of YARN
where we don't control the apps that run on the shared grid.

What problem are we trying to solve here? If we want better HDFS
performance and QOS for services then we want to give as much control over
the disk to HDFS rather than take it away. Short circuit reads leave a
gaping hole towards that end and making short circuit reads better and
easier to use makes that hole larger.

I am sorry for replying late and also because my response might be missing
historical perspectives that I am not aware of.

Bikas

-----Original Message-----
From: rarecactus@gmail.com [mailto:rarecactus@gmail.com] On Behalf Of
Colin McCabe
Sent: Sunday, February 17, 2013 1:49 PM
To: hdfs-dev@hadoop.apache.org
Subject: VOTE: HDFS-347 merge

Hi all,

I would like to merge the HDFS-347 branch back to trunk.  It's been under
intensive review and testing for several months.  The branch adds a lot of
new unit tests, and passes Jenkins as of 2/15 [1]

We have tested HDFS-347 with both random and sequential workloads. The
short-circuit case is substantially faster [2], and overall performance
looks very good.  This is especially encouraging given that the initial
goal of this work was to make security compatible with short-circuit local
reads, rather than to optimize the short-circuit code path.  We've also
stress-tested HDFS-347 on a number of clusters.

This iniial VOTE is to merge only into trunk.  Just as we have done with
our other recent merges, we will consider merging into branch-2 after the
code has been in trunk for few weeks.

Please cast your vote by EOD Sunday 2/24.

best,
Colin McCabe

[1]
https://issues.apache.org/jira/browse/HDFS-347?focusedCommentId=13579704&p
age=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comme
nt-13579704

[2]
https://issues.apache.org/jira/browse/HDFS-347?focusedCommentId=13551755&p
age=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comme
nt-13551755