hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0
Date Tue, 10 May 2016 16:44:33 GMT
On Mon, May 9, 2016 at 11:59 PM, Gary Helmling <ghelmling@gmail.com> wrote:
...


> To me, it seems much safer to actively try to push this upstream into HDFS
> right now, and still pointing to its optional, non-default use in HBase as
> a compelling story.  I don't understand why making it the default in 2.0 is
> necessary for this.  Do you really think it will make that big a difference
> for upstreaming?  Once it's actually in Hadoop and maintained, it seems
> like a no-brainer to make it the default.
>
>
The suggestion is that we make this new client the default now in master
branch so we have plenty of time to find any issues with the
implementation. We'd also enable it as the default because the improvement
is dramatic (performance, less moving parts, comprehensible, etc.) and we
think this async, lightweight, WAL-purposed client the way to go moving
forward. We'd also spare our users having to make a choice; if optional, if
they trip over the feature at all, they'll be wary enabling such a
fundamental afraid that it experimental.

Going this route we are taking on risk as you call out Gary but the
suggestion is that the benefits far outweigh the downsides (In mitigation,
I don't think we've ever run against HDFS free of reflection code though,
true, we are into a new level of violation with this client).

We are not arguing that it needs to be default to help get the async client
upstreamed into HDFS. Maybe it would help but going by the issue cited by
Duo below, it seems like there are a bunch of other concerns and dimensions
to be considered first; it may take a while for an async DFS client to land
(if at all). We should push on the upstreaming yes to close down the risk
you note, but let us not predicate our use of async WAL on it first being
committed to HDFS.

Thanks,
St.Ack







> On Mon, May 9, 2016 at 5:09 PM Stack <stack@duboce.net> wrote:
>
> > Any other suggestions/objections here? If not, will make the cut over in
> > next day or so.
> > Thanks,
> > St.Ack
> >
> > On Thu, May 5, 2016 at 10:02 PM, Stack <stack@duboce.net> wrote:
> >
> > > On Thu, May 5, 2016 at 7:39 PM, Yu Li <carp84@gmail.com> wrote:
> > >
> > >> Almost miss the party...
> > >>
> > >> bq. Do you think it worth to backport this feature to branch-1 and
> > release
> > >> it in the next 1.x release? This may introduce a compatibility issue
> as
> > >> said
> > >> in HBASE-14949 that we need HBASE-14949 to make sure that the rolling
> > >> upgrade
> > >> does not lose data...
> > >> From current perf data I think the effort is worthwhile, we already
> > >> started
> > >> some work here and will run it on production after some carefully
> > testing
> > >> (and of course, if the perf number confirmed, but I'm optimistic
> somehow
> > >> :-P). Regarding HBASE-14949, I guess a two-step rolling upgrade will
> > make
> > >> it work, right? (And I guess this will also be a question when we
> > upgrade
> > >> from 1.x to 2.0 later?)
> > >>
> > >>
> > > Or a clean shutdown and restart? Or a fresh install? I'd think backport
> > > would be fine if you have to enable it and it has warnings and is clear
> > on
> > > circumstances under which there could be dataloss.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >> btw, I'm +1 about making asyncfswal as default in 2.0 :-)
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >> On 6 May 2016 at 09:49, Ted Yu <yuzhihong@gmail.com> wrote:
> > >>
> > >> > Thanks for your effort, Duo.
> > >> >
> > >> > I am in favor of turning AsyncWAL as default in master branch.
> > >> >
> > >> > Cheers
> > >> >
> > >> > On Thu, May 5, 2016 at 6:03 PM, 张铎 <palomino219@gmail.com>
wrote:
> > >> >
> > >> > > Some progress.
> > >> > >
> > >> > > I have filed HBASE-15743 for the transparent encryption support,
> > >> > > and HBASE-15754 for the AES encryption UT. Now both of them are
> > >> resolved.
> > >> > > Let's resume the discussion here.
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > > 2016-05-03 10:09 GMT+08:00 张铎 <palomino219@gmail.com>:
> > >> > >
> > >> > > > Fine, will add the testcase.
> > >> > > >
> > >> > > > And for the RPC, we only implement a new client side DTP
here
> and
> > >> still
> > >> > > > use the original RPC.
> > >> > > >
> > >> > > > Thanks.
> > >> > > >
> > >> > > > 2016-05-03 3:20 GMT+08:00 Gary Helmling <ghelmling@gmail.com>:
> > >> > > >
> > >> > > >> On Fri, Apr 29, 2016 at 6:24 PM 张铎 <palomino219@gmail.com>
> > wrote:
> > >> > > >>
> > >> > > >> > Yes, it does. There is testcase that enumerates
all the
> > possible
> > >> > > >> protection
> > >> > > >> > level(authentication, integrity and privacy) and
encryption
> > >> > > >> algorithm(none,
> > >> > > >> > 3des, rc4).
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java
> > >> > > >> >
> > >> > > >> > I have also tested it in a secure
> cluster(hbase-2.0.0-SNAPSHOT
> > >> and
> > >> > > >> > hadoop-2.4.0).
> > >> > > >> >
> > >> > > >>
> > >> > > >> Thanks.  Can you add in support for testing with AES
> > >> > > >> (dfs.encrypt.data.transfer.cipher.suites=AES/CTR/NoPadding)?
> > This
> > >> is
> > >> > > only
> > >> > > >> available in Hadoop 2.6.0+, but I think is far more
likely to
> be
> > >> used
> > >> > in
> > >> > > >> production than 3des or rc4.
> > >> > > >
> > >> > > >
> > >> > > >> Also, have you been following HADOOP-10768?  That is
changing
> > >> Hadoop
> > >> > RPC
> > >> > > >> encryption negotiation to support more performant AES
wrapping,
> > >> > similar
> > >> > > to
> > >> > > >> what is now supported in the data transfer pipeline.
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message