hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张铎(Duo Zhang) <palomino...@gmail.com>
Subject Re: Moving 2.0 forward
Date Thu, 26 Oct 2017 09:44:29 GMT
Alpha4 aims to freeze the CP related API, not the public API. We break
everything for CP so we need to push out a stable(I hope) version to let CP
users learn how to implement their project on top of the new APIs.

For public API we have rules on how to keep compatible so I think it is
less hurt for users, beta1 is fine.

2017-10-26 17:27 GMT+08:00 Guanghao Zhang <zghaobac@gmail.com>:

> I saw beta == no new features, no API changes, just fixes. And I am working
> on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> 2.0-beta-1. But I thought this will introduce API change(deprecate some API
> and introduce new one). So should I change the fix versions to 2.0-alpha-4
> and finish it before we release 2.0-alpha-4?
>
> Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
> problem. Thanks.
>
> 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <palomino219@gmail.com>:
>
> > OK, skimmed,  we are in trouble! The in memory compaction just use the
> same
> > constructor with normal compaction to construct a StoreScanner, and use
> it
> > to do compaction...
> >
> > We have to provide several preXXX and postXXX for it, at least we should
> > allow user reset TTL and max versions, and also do filtering on the
> > scanner.
> >
> > Thanks.
> >
> >
> >
> > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <palomino219@gmail.com>:
> >
> > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> > in
> > > memory compaction.
> > >
> > > As Anoop suggested in HBASE-19001, we still need to give user the
> ability
> > > to extend the max versions and TTL config, so I plan to add back the
> > hooks
> > > above to let CP users can change the max versions and TTL of a ScanInfo
> > > object. But I'm not sure whether in memory compaction will also discard
> > > expired cells, if so then we are in trouble...
> > >
> > > Thanks.
> > >
> > > 2017-10-25 6:03 GMT+08:00 Stack <stack@duboce.net>:
> > >
> > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> > >> what remains to be done. Surfacing our thoughts here so you all clued
> > >> in....Or if you think otherwise, please speak up.
> > >>
> > >> We have ~13 issues to land:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> > two
> > >> are meta-issues that are about process which leaves 11.
> > >>
> > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > changed
> > >> API/semantic that we need to get out earlier rather than later.
> > >>
> > >> Once the above is merged, HBASE-13346, change of Filter method names
> to
> > >> mention Cell instead of KeyValue can land.
> > >>
> > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > >> IA.Private objects as params to MasterObserver... Hopefully this goes
> in
> > >> soon.
> > >>
> > >> Duo is hard at work on trackers for flush and compaction for CPs
> > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > >>
> > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> Duo
> > >> is
> > >> done w/ his work above.
> > >>
> > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> all
> > >> the
> > >> purges allowing CPs do direct calls against Regions in same Host.
> > >>
> > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >>
> > >> Another day or two?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <stack@duboce.net> wrote:
> > >>
> > >> >
> > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <elserj@apache.org>
> > wrote:
> > >> >
> > >> >> +1
> > >> >>
> > >> >> I was trying to work on helping out on the outstanding alpha-4
> stuff
> > >> last
> > >> >> week -- will be continuing to try to do the same this week.
> > >> >>
> > >> >> If you need any help, Stack, or if others need reviews where I
> > haven't
> > >> >> noticed on my own: feel free to @mention me.
> > >> >>
> > >> >>
> > >> > Thanks for the offer Josh. All items seem assigned and are being
> > >> actively
> > >> > worked on. If you get a moment, reviews by you (or anyone else)
> helps
> > >> move
> > >> > the process along.
> > >> >
> > >> > We need to merge in HBASE-18410 branch to pick up Filter
> improvements.
> > >> > Then HBASE-13346 can go in.
> > >> >
> > >> > You are already helping out on HBASE-18906, thanks. Looks like that
> > will
> > >> > be addressed by other alpha-4s about to land.
> > >> >
> > >> > St.Ack
> > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>
> > >> >>> (Reviving this thread)
> > >> >>>
> > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that
has
> the
> > >> >>> refactor of the Coprocessor API shutting down access to internals
> > >> marked
> > >> >>> InterfaceAudience.Private.
> > >> >>>
> > >> >>> The outstanding list is here:
> > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >> >>>
> > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>
> > >> >>> If issue, talk out loud on this thread. If you need a review
to
> land
> > >> an
> > >> >>> item, shout on the issue and here; we'll help you out.
> > >> >>>
> > >> >>> As is, items are coming along nicely I'd say. We need to merge
the
> > >> filter
> > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>
> > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and
help
> > them
> > >> >>> test
> > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence
our
> > >> >>> downstreamers know what to expect (or we discover what we
missed
> > >> BEFORE
> > >> >>> we
> > >> >>> beta-1).
> > >> >>>
> > >> >>> Thanks for time,
> > >> >>> S
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <stack@duboce.net>
wrote:
> > >> >>>
> > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> > be
> > >> >>>> time, if we all sprint, for the public-facing API fixes
to be
> done.
> > >> >>>>
> > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled
for
> > alpha3
> > >> but
> > >> >>>> it is plain that more time is needed (in spite of valiant
effort
> so
> > >> far
> > >> >>>> by
> > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> > >> >>>> theme is
> > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up
by the
> > >> following
> > >> >>>> week.
> > >> >>>>
> > >> >>>> We should then be ready for beta (beta == no new features,
no API
> > >> >>>> changes,
> > >> >>>> just fixes).
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>>
> > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <stack@duboce.net>
> wrote:
> > >> >>>>
> > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
vote on
> > it.
> > >> >>>>>
> > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API.
I hope to
> > get
> > >> a
> > >> >>>>> release out in the next week or so.
> > >> >>>>>
> > >> >>>>> I did a weeding of 2.0.0 issues over the last day.
If folks are
> > >> >>>>> interested in helping out, below are the items I think
we need
> > done
> > >> for
> > >> >>>>> alpha3 (below are at least 'Critical' status, are
API possibly
> > >> altering
> > >> >>>>> items, and are absent those JIRAs that are making
active
> progress,
> > >> >>>>> i.e. the
> > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
that
> needs
> > >> >>>>> doing is
> > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>
> > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between
branch-1
> and
> > >> >>>>> branch-2
> > >> >>>>> This is to do what Andrew did between 1.3 and 1.4
branches only
> do
> > >> it
> > >> >>>>> between branch-1 and branch-2.
> > >> >>>>>
> > >> >>>>> * HBASE-10462 Recategorize some of the client facing
Public /
> > >> Private
> > >> >>>>> interfaces
> > >> >>>>> This one is almost done. It could do with a finish,
attention to
> > the
> > >> >>>>> items in last comment, and then our codebase could
do with
> another
> > >> >>>>> sweep
> > >> >>>>> after the spirit of this issue since a bunch has gone
in since
> the
> > >> pass
> > >> >>>>> that was the basis of this issue.
> > >> >>>>>
> > >> >>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>> I was going to take a crack at this as part of the
revamp forced
> > by
> > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends
Guava's
> > Service'
> > >> >>>>> but if
> > >> >>>>> anyone else is interested, be my guest.
> > >> >>>>>
> > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>> Has a bunch of subtasks, some of which are being worked
on.
> Needs
> > >> >>>>> finishing.
> > >> >>>>>
> > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods
in
> Admin
> > >> and
> > >> >>>>> cleanup
> > >> >>>>> Needs a pass. Small issue I think. Could also look
at new
> > >> AsyncClient
> > >> >>>>> and
> > >> >>>>> make sure symmetry.
> > >> >>>>>
> > >> >>>>> * HBASE-15607 Remove PB references from Admin for
2.0
> > >> >>>>> Predicated on result of an ongoing DISCUSSION thread
but needs
> to
> > be
> > >> >>>>> done.
> > >> >>>>>
> > >> >>>>> Rolling upgrade will have implications for our API.
Would be
> good
> > to
> > >> >>>>> try
> > >> >>>>> it and figure what needs fixup (as said above, according
to
> trial
> > by
> > >> >>>>> Sean,
> > >> >>>>> we might not be too bad here):
> > >> >>>>> * HBASE-16060 1.x clients cannot access table state
talking to
> 2.0
> > >> >>>>> cluster
> > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility
for 2.x Master
> > and
> > >> >>>>> 1.x
> > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to
-2.
> > >> >>>>>
> > >> >>>>> * HBASE-17442 Move most of the replication related
classes to
> > >> >>>>> hbase-server package
> > >> >>>>> The above would be good to do generally but it may
make for
> > ripples
> > >> in
> > >> >>>>> API so would be good to do now.
> > >> >>>>>
> > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>> Balazs is working on this. The idea is that we avoid
adding two
> > new
> > >> >>>>> types
> > >> >>>>> to our API, two types that are nought but curtailed,
read-only
> > >> views on
> > >> >>>>> internals. Input if you have time appreciated.
> > >> >>>>>
> > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
to a
> > >> hbase2
> > >> >>>>> cluster; verify
> > >> >>>>> Esteban is looking at this one
> > >> >>>>>
> > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded
in core
> > >> >>>>> * HBASE-17143 Scan improvement
> > >> >>>>>
> > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>> implementations
> > >> >>>>> marked audience Private and returns implementations
rather than
> > >> >>>>> Interfaces.
> > >> >>>>> In a few locations, we allow returning an alternate
> implementation
> > >> >>>>> altogether which is probably something we don't want
a CP doing.
> > To
> > >> >>>>> that
> > >> >>>>> end, the following issues started by Duo and Anoop
need to be
> > taken
> > >> to
> > >> >>>>> the
> > >> >>>>> finish line; ideally they'd have an owner:
> > >> >>>>>
> > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
release
> <=
> > >> The
> > >> >>>>> umbrella issue.
> > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup
for CP
> expose
> > >> >>>>> * HBASE-16769 Deprecate/remove PB references from
MasterObserver
> > and
> > >> >>>>> RegionServerObserver
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Nice-to-haves:
> > >> >>>>>
> > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private
and remove
> > >> unused
> > >> >>>>> TimeRange constructors
> > >> >>>>>
> > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> references
> > >> >>>>> existing in the code
> > >> >>>>> This is the end of an old long-running project moving
up on to
> > Cell
> > >> >>>>> Interface. We think it is done but for a few little
items
> > >> (deprecate KV
> > >> >>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>
> > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation
is indeterminate;
> > >> needs
> > >> >>>>> fixing
> > >> >>>>>
> > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>
> > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>> * HBASE-14997
> > >> >>>>> Move compareOp and Comparators out of filter to client
package
> > >> >>>>>
> > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>
> > >> >>>>> What about:
> > >> >>>>>
> > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>
> > >> >>>>> If interested in any of the above and you need a legup,
just ask
> > in
> > >> the
> > >> >>>>> issue and I'll be by....
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> St.Ack
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net>
> wrote:
> > >> >>>>>
> > >> >>>>> Heads-up:
> > >> >>>>>>
> > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
Candidate.
> > Theme
> > >> is
> > >> >>>>>> updated dependencies, reliance on relocated popular
libs
> (guava,
> > >> >>>>>> netty,
> > >> >>>>>> protobuf), purge of checked-in generated src,
and
> > >> >>>>>> master-carries-no-regions
> > >> >>>>>> by default.
> > >> >>>>>>
> > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?).
Its
> theme
> > >> will
> > >> >>>>>> be
> > >> >>>>>> settling the APIs and compatibility (At first
blush, we are not
> > >> >>>>>> looking too
> > >> >>>>>> bad; our Sean ran some tests over weekend that
have hbase-1
> > client
> > >> >>>>>> running
> > >> >>>>>> against an hbase-2 cluster....). The Coprocessor
Interface
> revamp
> > >> >>>>>> should be
> > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather
than
> > >> >>>>>> Implementations, and
> > >> >>>>>> our shutdown of CPs accessing classes in hbase
marked
> > >> >>>>>> InterfaceAudience).
> > >> >>>>>> We'll also have purged thirdparty classes from
our API; e.g.
> > guava
> > >> >>>>>> 0.12
> > >> >>>>>> Service showing through in our replication API
and protobufs in
> > >> Admin
> > >> >>>>>> Interface. On alpha3, we will have to do a bunch
of outreach to
> > >> make
> > >> >>>>>> sure
> > >> >>>>>> our downstreamers are up on what is coming down
the pipe.
> > >> >>>>>>
> > >> >>>>>> Beta1 in mid-September?
> > >> >>>>>>
> > >> >>>>>> I encourage you to check out the items marked
for hbase2:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> > >> Edit
> > >> >>>>>> as
> > >> >>>>>> you see appropriate. Punt if you know the JIRA
will not get any
> > >> >>>>>> attention
> > >> >>>>>> in next month or so.
> > >> >>>>>>
> > >> >>>>>> A bunch of issues marked blocker are unassigned.
I'll leave
> them
> > >> as is
> > >> >>>>>> another while but I'll boot them soon.
> > >> >>>>>>
> > >> >>>>>> While I have your attention:
> > >> >>>>>>
> > >> >>>>>> + I think we should leave thrift version at 0.9.3.
Moving hbase
> > >> thrift
> > >> >>>>>> to 0.10.0 will break existing clients. The change
is easy
> enough
> > if
> > >> >>>>>> folks
> > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to
get to 1.0 first
> > >> >>>>>> (0.98?).
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <stack@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser
<elserj@apache.org
> >
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>
> > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh
Elser<
> elserj@apache.org
> > >
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> ...
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I like the idea of this also hitting
2.0 as it would make
> the
> > >> >>>>>>>>>> feature a
> > >> >>>>>>>>>> bit more "real", but am obviously
a little nervous (I have
> no
> > >> >>>>>>>>>> reason
> > >> >>>>>>>>>> to be
> > >> >>>>>>>>>> nervous though). I am pretty happy
with the feature in
> terms
> > of
> > >> >>>>>>>>>> how
> > >> >>>>>>>>>> much it
> > >> >>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Sounds good to me. Whats involved?
Backport? If so, +1
> Josh.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> Last think on space quota says that
need doc too. See 'Space
> > >> >>>>>>>>> Quota' in
> > >> >>>>>>>>> here:
> > >> >>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>> Does this little section need an update
Josh?
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>> Yep, just a couple of cherry-picks. Good
test coverage and
> some
> > >> docs
> > >> >>>>>>>> already included for 17748.  Happy to
put that on my plate if
> > >> >>>>>>>> you're good
> > >> >>>>>>>> with it. I can reasonably assume that
no one is against it :)
> > >> >>>>>>>>
> > >> >>>>>>>> I think I had knocked out docs for the
"phase 1" stuff before
> > we
> > >> >>>>>>>> merged it in from the original feature
branch. I'll double
> > check
> > >> >>>>>>>> and update
> > >> >>>>>>>> the gdoc. Perhaps this was just a timing
thing.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Thanks Josh,
> > >> >>>>>>> S
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message