Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <CADfYSXSQhurpPW-EinW_wzw+kx1Lzxd+6p25_7_RRXKqkQ8K=w@mail.gmail.com>
References: <CAAAYAnNPaD3eHHYBVAtZ9cc3GNqJ4iCMW1qc458Zm7c1r58obg@mail.gmail.com>
	<CAGHyZ6K81JWut-C1c0e-kF9CM7W9p3vi0cT4O=FAyv+L1qYVfw@mail.gmail.com>
	<CAAAYAnOv2prLc2hqNGBqEoog0ebDbYK_0kWYkK58kmcifh_UnQ@mail.gmail.com>
	<CALte62wfZk2yOUHyDFLPgnX=hEyLqbzKJoekkivKgEvx6H8AXQ@mail.gmail.com>
	<CAAAYAnPG9GUczdOEg8POa_KdWePsmYpDevUfOBiWgqsm1SBjUg@mail.gmail.com>
	<CALte62ytSUqPRDcNtJqDrXsq6fMYkt1BXPc5b5T92kaN0AtW7g@mail.gmail.com>
	<CADcMMgFO9CwOJhouUHD2Z+iPK21TWf4-pPnA4MTu2gvZ=Bt9Ew@mail.gmail.com>
	<CAAAYAnNtBOWeA181sTy_vkWmWYWMHSAAOrFfs1x1v9-7NUJwSA@mail.gmail.com>
	<CADcMMgE8pLCD+=jgi27F2hSLYZyE+6jMxCOsDfvFEfqJkJRb3w@mail.gmail.com>
	<CADfYSXQFPxtSa5Dvk3TbdE+0+=k7eSFrK0TRkNqjKGP4fnxktQ@mail.gmail.com>
	<CAAAYAnMNObWXuBohnMV79AZqcpPZa7KKCRbX7gt8c647apAPsQ@mail.gmail.com>
	<CADfYSXRUNndEuRJ5GDqBH_roUD++uWyO=GAea-FoXX3xubtGeg@mail.gmail.com>
	<CAAAYAnNO95pFSyU_4q18EQ68Nnj9aNKM+==9vHu=0fWB0d5eCw@mail.gmail.com>
	<CAAAYAnObTcSK+=y35r=Pv0h73VR9X+4qGoTe-PU=SREh_N98kQ@mail.gmail.com>
	<CALte62wZbEpHhZRFpAhyBPQgZo=i0jGF6Wcr1EsX6RK=drZkKw@mail.gmail.com>
	<CAM7-19+Gag6yNVuusNrC8cevSySqatT6cE3xdVBXdQff_ndJ=w@mail.gmail.com>
	<CADcMMgEkzOBGq4BygpJngbrukkzPHDG1OGbhRrMjnzP_Hmgsww@mail.gmail.com>
	<CADcMMgE8VKMqVccccVyzKejT1pa6a24vFMkDUrHGX_3KmxNGoA@mail.gmail.com>
	<CADfYSXSt8OKHb59d+FfAH8LUMu7pMBSs3smd_D4nzjJe2==dYA@mail.gmail.com>
	<CADcMMgEQApqZVZQmSuV4oQe+hs1eT_i+HRGmk6OVHiFcZTA6DQ@mail.gmail.com>
	<CADfYSXSQhurpPW-EinW_wzw+kx1Lzxd+6p25_7_RRXKqkQ8K=w@mail.gmail.com>
Date: Tue, 10 May 2016 12:04:49 -0700
Message-ID: <CADcMMgG3ts-OWgCcs1gvf+gc075K0=co-LcHtr0fqBUBoN=Z=A@mail.gmail.com>
Subject: Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0
From: Stack <stack@duboce.net>
To: HBase Dev List <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a113cf15269d88a053281992a
archived-at: Tue, 10 May 2016 19:04:56 -0000

--001a113cf15269d88a053281992a
Content-Type: text/plain; charset=UTF-8

On Tue, May 10, 2016 at 10:39 AM, Gary Helmling <ghelmling@gmail.com> wrote:

> >
> > The suggestion is that we make this new client the default now in master
> > branch so we have plenty of time to find any issues with the
> > implementation. We'd also enable it as the default because the
> improvement
> > is dramatic (performance, less moving parts, comprehensible, etc.) and we
> > think this async, lightweight, WAL-purposed client the way to go moving
> > forward. We'd also spare our users having to make a choice; if optional,
> if
> > they trip over the feature at all, they'll be wary enabling such a
> > fundamental afraid that it experimental.
> >
> >
> To me, having the opportunity to shake out issues sounds like an argument
> for making it experimental, not making it the default.


I was trying to avoid the below oft-repeated pattern at least for the case
of critical developments:

+ New feature arrives after much work by developer, reviewers and testers
accompanied by fanfare (blog, talks).
+ Developers and reviewers move on after getting it committed or it gets
hacked into a deploy so it works in a frankenstein form
+ It sits in our code base across one or more releases marked as optional,
'experimental'
+ The 'experimental' bleamish discourages its exercise by users
+ The feature lags, rots
+ Or, the odd time, we go ahead and enable it as default in spite of the
fact it was never tried when experimental.

Distributed Log Replay sat in hbase across a few major versions. Only when
the threat of our making an actual release with it on by default did it get
serious attention where it was found flawed and is now being actively
purged. This was after it made it past reviews, multiple attempts at
testing at scale, and so on; i.e. we'd done it all by the book. The time in
an 'experimental' state added nothing.


> I think that
> features we're rolling out to our users actively enabled should have some
> level of confidence that the issues are already shaken out.  I'm interested
> in testing this out myself, but personally I would want to test this by
> actively enabling it and not just having it show up unexpectedly in a new
> release.
>
> Maybe this is mitigated by the reality that a 2.0 release is not going to
> happen for quite a while.  But this also becomes a self-fulfilling prophecy
> if we continue to make disruptive changes in master.
>
> Yes. 2.0 is a bit out there so we have some time to iron out issues is the
thought. Yes, it could push out delivery of 2.0.


> > Going this route we are taking on risk as you call out Gary but the
> > suggestion is that the benefits far outweigh the downsides (In
> mitigation,
> > I don't think we've ever run against HDFS free of reflection code though,
> > true, we are into a new level of violation with this client).
> >
> >
> I'm not trying to argue for a perfect world. :)
>
> But I do think there is a big difference in some of our other past use of
> reflection to ride over changes in public APIs vs. reaching in to private
> fields in private annotated classes.  What would we say to a coprocessor
> that did the same with HBase?
>

You have a point. We'd break the dependent w/o regard as HDFS will do to
this feature until it gets pushed upstream.


> > We are not arguing that it needs to be default to help get the async
> client
> > upstreamed into HDFS. Maybe it would help but going by the issue cited by
> > Duo below, it seems like there are a bunch of other concerns and
> dimensions
> > to be considered first; it may take a while for an async DFS client to
> land
> > (if at all). We should push on the upstreaming yes to close down the risk
> > you note, but let us not predicate our use of async WAL on it first being
> > committed to HDFS.
> >
> >
> I don't think our ability to use the async WAL as an optional feature
> should be predicated on inclusion in HDFS.  But I do think our use of it as
> the default, and all the continuing support that that implies should be.
> That is where we disagree.
>
> I don't think we're really making progress with the discussion here.  I
> don't agree with the arguments being put forward, but it seems like I'm in
> the minority.
>

I think the discussion here has been helpful. Holes have been found (and
plugged), the risk involved has gotten a good airing out here on dev, and
in spite of the back and forth, one of our experts in good standing is
still against it being on by default.

If you are not down w/ the arguments, I'd be fine not making it the default.
St.Ack

--001a113cf15269d88a053281992a--