Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CADfYSXSM5hpBuhvxuGves_zvZaUG7ubzVN6jRLjQFem-ERqMBg@mail.gmail.com>
References: <CAAAYAnNPaD3eHHYBVAtZ9cc3GNqJ4iCMW1qc458Zm7c1r58obg@mail.gmail.com>
	<CAGHyZ6K81JWut-C1c0e-kF9CM7W9p3vi0cT4O=FAyv+L1qYVfw@mail.gmail.com>
	<CAAAYAnOv2prLc2hqNGBqEoog0ebDbYK_0kWYkK58kmcifh_UnQ@mail.gmail.com>
	<CALte62wfZk2yOUHyDFLPgnX=hEyLqbzKJoekkivKgEvx6H8AXQ@mail.gmail.com>
	<CAAAYAnPG9GUczdOEg8POa_KdWePsmYpDevUfOBiWgqsm1SBjUg@mail.gmail.com>
	<CALte62ytSUqPRDcNtJqDrXsq6fMYkt1BXPc5b5T92kaN0AtW7g@mail.gmail.com>
	<CADcMMgFO9CwOJhouUHD2Z+iPK21TWf4-pPnA4MTu2gvZ=Bt9Ew@mail.gmail.com>
	<CAAAYAnNtBOWeA181sTy_vkWmWYWMHSAAOrFfs1x1v9-7NUJwSA@mail.gmail.com>
	<CADcMMgE8pLCD+=jgi27F2hSLYZyE+6jMxCOsDfvFEfqJkJRb3w@mail.gmail.com>
	<CADfYSXQFPxtSa5Dvk3TbdE+0+=k7eSFrK0TRkNqjKGP4fnxktQ@mail.gmail.com>
	<CAAAYAnMNObWXuBohnMV79AZqcpPZa7KKCRbX7gt8c647apAPsQ@mail.gmail.com>
	<CADfYSXRUNndEuRJ5GDqBH_roUD++uWyO=GAea-FoXX3xubtGeg@mail.gmail.com>
	<CAAAYAnNO95pFSyU_4q18EQ68Nnj9aNKM+==9vHu=0fWB0d5eCw@mail.gmail.com>
	<CAAAYAnObTcSK+=y35r=Pv0h73VR9X+4qGoTe-PU=SREh_N98kQ@mail.gmail.com>
	<CALte62wZbEpHhZRFpAhyBPQgZo=i0jGF6Wcr1EsX6RK=drZkKw@mail.gmail.com>
	<CAM7-19+Gag6yNVuusNrC8cevSySqatT6cE3xdVBXdQff_ndJ=w@mail.gmail.com>
	<CADcMMgEkzOBGq4BygpJngbrukkzPHDG1OGbhRrMjnzP_Hmgsww@mail.gmail.com>
	<CADcMMgE8VKMqVccccVyzKejT1pa6a24vFMkDUrHGX_3KmxNGoA@mail.gmail.com>
	<CADfYSXSt8OKHb59d+FfAH8LUMu7pMBSs3smd_D4nzjJe2==dYA@mail.gmail.com>
	<CAAAYAnMmUjodNsSB_sJ7_F1GjWoG1NomaOOY8_ZGcNatKAW4DA@mail.gmail.com>
	<CADfYSXSM5hpBuhvxuGves_zvZaUG7ubzVN6jRLjQFem-ERqMBg@mail.gmail.com>
Date: Wed, 11 May 2016 07:16:28 +0800
Message-ID: <CAAAYAnPr9vQUc-F0qH0Q8+ywJoPUms9irvi87x9jyTrZkzfjXw@mail.gmail.com>
Subject: Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0
From: =?UTF-8?B?5byg6ZOO?= <palomino219@gmail.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a1142904468731d0532851dd6
archived-at: Tue, 10 May 2016 23:16:35 -0000

--001a1142904468731d0532851dd6
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

See HDFS-223 and HDFS-916. There are plenty of issues related. The most
important thing is that we need a suitable api and there is an asynchronous
file system proposal in HADOOP-12910 which does not fit our requirements so
I need to stop it being committed first...

And a default choice in a later HBase version means that this feature is
definitely needed in HDFS and will be well tested even before it go into
the HDFS code base. An experimental feature is always experimental. Think
of DLR. For per CF flush, the data loss issue had been found before
releasing 1.2 in which version it is on by default. And we also experienced
this recently when backporting the scan heartbeat feature into our
internal branch. Lots of small bugs though the code has been there for
months...

Thanks.

2016=E5=B9=B45=E6=9C=8811=E6=97=A5=E6=98=9F=E6=9C=9F=E4=B8=89=EF=BC=8CGary =
Helmling <ghelmling@gmail.com> =E5=86=99=E9=81=93=EF=BC=9A

> >
> > Yeah the 'push to upstream' work has been started already. See here
> >
> > https://issues.apache.org/jira/browse/HADOOP-12910
> >
> > But it is much harder to push code into HDFS than HBase. It is the core
> of
> > all hadoop systems and I do not have many contacts in the hdfs
> community...
> >
> >
> Yes, I'm familiar with the difficulty of getting even relatively small
> change in to HDFS.
>
> Is HADOOP-12910 really the right issue for this?  There is some good
> discussion there, but it looks like it's primarily motivated by doing lar=
ge
> batches of async renames.  Isn't our issue more that we want a whole new
> OutputStream implementation doing fan out instead of the regular pipeline
> replication?
>
> HDFS-9924 seems like it might be the better umbrella for this.  Maybe it
> would be better to create a new top level issue and link it there and
> comment in HDFS-9924 to try to get some traction.
>
>
> > And it is more convincing if we make it default as it means that we wil=
l
> > keep maintaining the code rather than make it stale and unstable.
> >
> >
> I would agree with this reasoning if we were talking about making an
> implementation inside HDFS the default.  That would indicate commitment t=
o
> contribute to and maintain the HDFS implementation.  Making a separate co=
de
> copy living in HBase the default I don't think really means anything for
> HDFS.
>
> The fact that this already needs to be updated for 2.8 just reinforces th=
at
> we're going to see maintainability issues with this.
>
> Again, I appreciate all of the work that has gone in to this feature and
> the big performance improvements it shows, but I think the sequencing of
> events here is going to ultimately cause us and our users more pain than =
is
> necessary.
>

--001a1142904468731d0532851dd6--