hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Moving 2.0 forward
Date Thu, 26 Oct 2017 16:14:50 GMT
On Thu, Oct 26, 2017 at 2:27 AM, Guanghao Zhang <zghaobac@gmail.com> wrote:

> I saw beta == no new features, no API changes, just fixes. And I am working
> on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> 2.0-beta-1. But I thought this will introduce API change(deprecate some API
> and introduce new one). So should I change the fix versions to 2.0-alpha-4
> and finish it before we release 2.0-alpha-4?
>
> Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
> problem. Thanks.
>
>
Yeah, we were supposed to be done w/ public API changes when we shipped
alpha-3 (smile).

For Admin Interface, semantic versioning applies as per Duo comment. I see
some polish still being done on the Admin/Table APIs (removal of deprecated
methods, alignment of Admin and AsyncAdmin APIs). Hard to say no to this
clean up on our most public facing Interfaces.

When you think you'd be done w/ the sync of Admin and AsyncAdmin
Interfaces? Are the changes to Admin or to AsyncAdmin?

Thank you Guanghao Zhang,

St.Ack


> 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <palomino219@gmail.com>:
>
> > OK, skimmed,  we are in trouble! The in memory compaction just use the
> same
> > constructor with normal compaction to construct a StoreScanner, and use
> it
> > to do compaction...
> >
> > We have to provide several preXXX and postXXX for it, at least we should
> > allow user reset TTL and max versions, and also do filtering on the
> > scanner.
> >
> > Thanks.
> >
> >
> >
> > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <palomino219@gmail.com>:
> >
> > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> > in
> > > memory compaction.
> > >
> > > As Anoop suggested in HBASE-19001, we still need to give user the
> ability
> > > to extend the max versions and TTL config, so I plan to add back the
> > hooks
> > > above to let CP users can change the max versions and TTL of a ScanInfo
> > > object. But I'm not sure whether in memory compaction will also discard
> > > expired cells, if so then we are in trouble...
> > >
> > > Thanks.
> > >
> > > 2017-10-25 6:03 GMT+08:00 Stack <stack@duboce.net>:
> > >
> > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> > >> what remains to be done. Surfacing our thoughts here so you all clued
> > >> in....Or if you think otherwise, please speak up.
> > >>
> > >> We have ~13 issues to land:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> > two
> > >> are meta-issues that are about process which leaves 11.
> > >>
> > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > changed
> > >> API/semantic that we need to get out earlier rather than later.
> > >>
> > >> Once the above is merged, HBASE-13346, change of Filter method names
> to
> > >> mention Cell instead of KeyValue can land.
> > >>
> > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > >> IA.Private objects as params to MasterObserver... Hopefully this goes
> in
> > >> soon.
> > >>
> > >> Duo is hard at work on trackers for flush and compaction for CPs
> > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > >>
> > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> Duo
> > >> is
> > >> done w/ his work above.
> > >>
> > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> all
> > >> the
> > >> purges allowing CPs do direct calls against Regions in same Host.
> > >>
> > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >>
> > >> Another day or two?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <stack@duboce.net> wrote:
> > >>
> > >> >
> > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <elserj@apache.org>
> > wrote:
> > >> >
> > >> >> +1
> > >> >>
> > >> >> I was trying to work on helping out on the outstanding alpha-4
> stuff
> > >> last
> > >> >> week -- will be continuing to try to do the same this week.
> > >> >>
> > >> >> If you need any help, Stack, or if others need reviews where I
> > haven't
> > >> >> noticed on my own: feel free to @mention me.
> > >> >>
> > >> >>
> > >> > Thanks for the offer Josh. All items seem assigned and are being
> > >> actively
> > >> > worked on. If you get a moment, reviews by you (or anyone else)
> helps
> > >> move
> > >> > the process along.
> > >> >
> > >> > We need to merge in HBASE-18410 branch to pick up Filter
> improvements.
> > >> > Then HBASE-13346 can go in.
> > >> >
> > >> > You are already helping out on HBASE-18906, thanks. Looks like that
> > will
> > >> > be addressed by other alpha-4s about to land.
> > >> >
> > >> > St.Ack
> > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>
> > >> >>> (Reviving this thread)
> > >> >>>
> > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that
has
> the
> > >> >>> refactor of the Coprocessor API shutting down access to internals
> > >> marked
> > >> >>> InterfaceAudience.Private.
> > >> >>>
> > >> >>> The outstanding list is here:
> > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >> >>>
> > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>
> > >> >>> If issue, talk out loud on this thread. If you need a review
to
> land
> > >> an
> > >> >>> item, shout on the issue and here; we'll help you out.
> > >> >>>
> > >> >>> As is, items are coming along nicely I'd say. We need to merge
the
> > >> filter
> > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>
> > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and
help
> > them
> > >> >>> test
> > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence
our
> > >> >>> downstreamers know what to expect (or we discover what we
missed
> > >> BEFORE
> > >> >>> we
> > >> >>> beta-1).
> > >> >>>
> > >> >>> Thanks for time,
> > >> >>> S
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <stack@duboce.net>
wrote:
> > >> >>>
> > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> > be
> > >> >>>> time, if we all sprint, for the public-facing API fixes
to be
> done.
> > >> >>>>
> > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled
for
> > alpha3
> > >> but
> > >> >>>> it is plain that more time is needed (in spite of valiant
effort
> so
> > >> far
> > >> >>>> by
> > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> > >> >>>> theme is
> > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up
by the
> > >> following
> > >> >>>> week.
> > >> >>>>
> > >> >>>> We should then be ready for beta (beta == no new features,
no API
> > >> >>>> changes,
> > >> >>>> just fixes).
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>>
> > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <stack@duboce.net>
> wrote:
> > >> >>>>
> > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
vote on
> > it.
> > >> >>>>>
> > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API.
I hope to
> > get
> > >> a
> > >> >>>>> release out in the next week or so.
> > >> >>>>>
> > >> >>>>> I did a weeding of 2.0.0 issues over the last day.
If folks are
> > >> >>>>> interested in helping out, below are the items I think
we need
> > done
> > >> for
> > >> >>>>> alpha3 (below are at least 'Critical' status, are
API possibly
> > >> altering
> > >> >>>>> items, and are absent those JIRAs that are making
active
> progress,
> > >> >>>>> i.e. the
> > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
that
> needs
> > >> >>>>> doing is
> > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>
> > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between
branch-1
> and
> > >> >>>>> branch-2
> > >> >>>>> This is to do what Andrew did between 1.3 and 1.4
branches only
> do
> > >> it
> > >> >>>>> between branch-1 and branch-2.
> > >> >>>>>
> > >> >>>>> * HBASE-10462 Recategorize some of the client facing
Public /
> > >> Private
> > >> >>>>> interfaces
> > >> >>>>> This one is almost done. It could do with a finish,
attention to
> > the
> > >> >>>>> items in last comment, and then our codebase could
do with
> another
> > >> >>>>> sweep
> > >> >>>>> after the spirit of this issue since a bunch has gone
in since
> the
> > >> pass
> > >> >>>>> that was the basis of this issue.
> > >> >>>>>
> > >> >>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>> I was going to take a crack at this as part of the
revamp forced
> > by
> > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends
Guava's
> > Service'
> > >> >>>>> but if
> > >> >>>>> anyone else is interested, be my guest.
> > >> >>>>>
> > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>> Has a bunch of subtasks, some of which are being worked
on.
> Needs
> > >> >>>>> finishing.
> > >> >>>>>
> > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods
in
> Admin
> > >> and
> > >> >>>>> cleanup
> > >> >>>>> Needs a pass. Small issue I think. Could also look
at new
> > >> AsyncClient
> > >> >>>>> and
> > >> >>>>> make sure symmetry.
> > >> >>>>>
> > >> >>>>> * HBASE-15607 Remove PB references from Admin for
2.0
> > >> >>>>> Predicated on result of an ongoing DISCUSSION thread
but needs
> to
> > be
> > >> >>>>> done.
> > >> >>>>>
> > >> >>>>> Rolling upgrade will have implications for our API.
Would be
> good
> > to
> > >> >>>>> try
> > >> >>>>> it and figure what needs fixup (as said above, according
to
> trial
> > by
> > >> >>>>> Sean,
> > >> >>>>> we might not be too bad here):
> > >> >>>>> * HBASE-16060 1.x clients cannot access table state
talking to
> 2.0
> > >> >>>>> cluster
> > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility
for 2.x Master
> > and
> > >> >>>>> 1.x
> > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to
-2.
> > >> >>>>>
> > >> >>>>> * HBASE-17442 Move most of the replication related
classes to
> > >> >>>>> hbase-server package
> > >> >>>>> The above would be good to do generally but it may
make for
> > ripples
> > >> in
> > >> >>>>> API so would be good to do now.
> > >> >>>>>
> > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>> Balazs is working on this. The idea is that we avoid
adding two
> > new
> > >> >>>>> types
> > >> >>>>> to our API, two types that are nought but curtailed,
read-only
> > >> views on
> > >> >>>>> internals. Input if you have time appreciated.
> > >> >>>>>
> > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
to a
> > >> hbase2
> > >> >>>>> cluster; verify
> > >> >>>>> Esteban is looking at this one
> > >> >>>>>
> > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded
in core
> > >> >>>>> * HBASE-17143 Scan improvement
> > >> >>>>>
> > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>> implementations
> > >> >>>>> marked audience Private and returns implementations
rather than
> > >> >>>>> Interfaces.
> > >> >>>>> In a few locations, we allow returning an alternate
> implementation
> > >> >>>>> altogether which is probably something we don't want
a CP doing.
> > To
> > >> >>>>> that
> > >> >>>>> end, the following issues started by Duo and Anoop
need to be
> > taken
> > >> to
> > >> >>>>> the
> > >> >>>>> finish line; ideally they'd have an owner:
> > >> >>>>>
> > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
release
> <=
> > >> The
> > >> >>>>> umbrella issue.
> > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup
for CP
> expose
> > >> >>>>> * HBASE-16769 Deprecate/remove PB references from
MasterObserver
> > and
> > >> >>>>> RegionServerObserver
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Nice-to-haves:
> > >> >>>>>
> > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private
and remove
> > >> unused
> > >> >>>>> TimeRange constructors
> > >> >>>>>
> > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> references
> > >> >>>>> existing in the code
> > >> >>>>> This is the end of an old long-running project moving
up on to
> > Cell
> > >> >>>>> Interface. We think it is done but for a few little
items
> > >> (deprecate KV
> > >> >>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>
> > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation
is indeterminate;
> > >> needs
> > >> >>>>> fixing
> > >> >>>>>
> > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>
> > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>> * HBASE-14997
> > >> >>>>> Move compareOp and Comparators out of filter to client
package
> > >> >>>>>
> > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>
> > >> >>>>> What about:
> > >> >>>>>
> > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>
> > >> >>>>> If interested in any of the above and you need a legup,
just ask
> > in
> > >> the
> > >> >>>>> issue and I'll be by....
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> St.Ack
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net>
> wrote:
> > >> >>>>>
> > >> >>>>> Heads-up:
> > >> >>>>>>
> > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
Candidate.
> > Theme
> > >> is
> > >> >>>>>> updated dependencies, reliance on relocated popular
libs
> (guava,
> > >> >>>>>> netty,
> > >> >>>>>> protobuf), purge of checked-in generated src,
and
> > >> >>>>>> master-carries-no-regions
> > >> >>>>>> by default.
> > >> >>>>>>
> > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?).
Its
> theme
> > >> will
> > >> >>>>>> be
> > >> >>>>>> settling the APIs and compatibility (At first
blush, we are not
> > >> >>>>>> looking too
> > >> >>>>>> bad; our Sean ran some tests over weekend that
have hbase-1
> > client
> > >> >>>>>> running
> > >> >>>>>> against an hbase-2 cluster....). The Coprocessor
Interface
> revamp
> > >> >>>>>> should be
> > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather
than
> > >> >>>>>> Implementations, and
> > >> >>>>>> our shutdown of CPs accessing classes in hbase
marked
> > >> >>>>>> InterfaceAudience).
> > >> >>>>>> We'll also have purged thirdparty classes from
our API; e.g.
> > guava
> > >> >>>>>> 0.12
> > >> >>>>>> Service showing through in our replication API
and protobufs in
> > >> Admin
> > >> >>>>>> Interface. On alpha3, we will have to do a bunch
of outreach to
> > >> make
> > >> >>>>>> sure
> > >> >>>>>> our downstreamers are up on what is coming down
the pipe.
> > >> >>>>>>
> > >> >>>>>> Beta1 in mid-September?
> > >> >>>>>>
> > >> >>>>>> I encourage you to check out the items marked
for hbase2:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> > >> Edit
> > >> >>>>>> as
> > >> >>>>>> you see appropriate. Punt if you know the JIRA
will not get any
> > >> >>>>>> attention
> > >> >>>>>> in next month or so.
> > >> >>>>>>
> > >> >>>>>> A bunch of issues marked blocker are unassigned.
I'll leave
> them
> > >> as is
> > >> >>>>>> another while but I'll boot them soon.
> > >> >>>>>>
> > >> >>>>>> While I have your attention:
> > >> >>>>>>
> > >> >>>>>> + I think we should leave thrift version at 0.9.3.
Moving hbase
> > >> thrift
> > >> >>>>>> to 0.10.0 will break existing clients. The change
is easy
> enough
> > if
> > >> >>>>>> folks
> > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to
get to 1.0 first
> > >> >>>>>> (0.98?).
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <stack@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser
<elserj@apache.org
> >
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>
> > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh
Elser<
> elserj@apache.org
> > >
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> ...
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I like the idea of this also hitting
2.0 as it would make
> the
> > >> >>>>>>>>>> feature a
> > >> >>>>>>>>>> bit more "real", but am obviously
a little nervous (I have
> no
> > >> >>>>>>>>>> reason
> > >> >>>>>>>>>> to be
> > >> >>>>>>>>>> nervous though). I am pretty happy
with the feature in
> terms
> > of
> > >> >>>>>>>>>> how
> > >> >>>>>>>>>> much it
> > >> >>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Sounds good to me. Whats involved?
Backport? If so, +1
> Josh.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> Last think on space quota says that
need doc too. See 'Space
> > >> >>>>>>>>> Quota' in
> > >> >>>>>>>>> here:
> > >> >>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>> Does this little section need an update
Josh?
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>> Yep, just a couple of cherry-picks. Good
test coverage and
> some
> > >> docs
> > >> >>>>>>>> already included for 17748.  Happy to
put that on my plate if
> > >> >>>>>>>> you're good
> > >> >>>>>>>> with it. I can reasonably assume that
no one is against it :)
> > >> >>>>>>>>
> > >> >>>>>>>> I think I had knocked out docs for the
"phase 1" stuff before
> > we
> > >> >>>>>>>> merged it in from the original feature
branch. I'll double
> > check
> > >> >>>>>>>> and update
> > >> >>>>>>>> the gdoc. Perhaps this was just a timing
thing.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Thanks Josh,
> > >> >>>>>>> S
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message