hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张铎(Duo Zhang) <palomino...@gmail.com>
Subject Re: Moving 2.0 forward
Date Thu, 26 Oct 2017 01:41:38 GMT
When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
HBASE-19033, I found that there maybe a hole in our CP hools. It is the in
memory compaction.

As Anoop suggested in HBASE-19001, we still need to give user the ability
to extend the max versions and TTL config, so I plan to add back the hooks
above to let CP users can change the max versions and TTL of a ScanInfo
object. But I'm not sure whether in memory compaction will also discard
expired cells, if so then we are in trouble...

Thanks.

2017-10-25 6:03 GMT+08:00 Stack <stack@duboce.net>:

> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
> what remains to be done. Surfacing our thoughts here so you all clued
> in....Or if you think otherwise, please speak up.
>
> We have ~13 issues to land:
> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
> are meta-issues that are about process which leaves 11.
>
> Duo and Zheng Hu are to merge the FilterList fixes improvements
> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
> API/semantic that we need to get out earlier rather than later.
>
> Once the above is merged, HBASE-13346, change of Filter method names to
> mention Cell instead of KeyValue can land.
>
> HBASE-199048 needs a review (Anoop will probably do it), removing
> IA.Private objects as params to MasterObserver... Hopefully this goes in
> soon.
>
> Duo is hard at work on trackers for flush and compaction for CPs
> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>
> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
> done w/ his work above.
>
> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
> purges allowing CPs do direct calls against Regions in same Host.
>
> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>
> Another day or two?
>
> St.Ack
>
>
>
> On Mon, Oct 23, 2017 at 2:52 PM, Stack <stack@duboce.net> wrote:
>
> >
> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <elserj@apache.org> wrote:
> >
> >> +1
> >>
> >> I was trying to work on helping out on the outstanding alpha-4 stuff
> last
> >> week -- will be continuing to try to do the same this week.
> >>
> >> If you need any help, Stack, or if others need reviews where I haven't
> >> noticed on my own: feel free to @mention me.
> >>
> >>
> > Thanks for the offer Josh. All items seem assigned and are being actively
> > worked on. If you get a moment, reviews by you (or anyone else) helps
> move
> > the process along.
> >
> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> > Then HBASE-13346 can go in.
> >
> > You are already helping out on HBASE-18906, thanks. Looks like that will
> > be addressed by other alpha-4s about to land.
> >
> > St.Ack
> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >> On 10/23/17 12:53 PM, Stack wrote:
> >>
> >>> (Reviving this thread)
> >>>
> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >>> refactor of the Coprocessor API shutting down access to internals
> marked
> >>> InterfaceAudience.Private.
> >>>
> >>> The outstanding list is here:
> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>>
> >>> Please push in anything marked alpha-4 that belongs to you.
> >>>
> >>> If issue, talk out loud on this thread. If you need a review to land an
> >>> item, shout on the issue and here; we'll help you out.
> >>>
> >>> As is, items are coming along nicely I'd say. We need to merge the
> filter
> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>>
> >>> Post alpha-4, we'll have to hunt down our downstreamers and help them
> >>> test
> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> >>> downstreamers know what to expect (or we discover what we missed BEFORE
> >>> we
> >>> beta-1).
> >>>
> >>> Thanks for time,
> >>> S
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <stack@duboce.net> wrote:
> >>>
> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> >>>> time, if we all sprint, for the public-facing API fixes to be done.
> >>>>
> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> but
> >>>> it is plain that more time is needed (in spite of valiant effort so
> far
> >>>> by
> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> >>>> theme is
> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> following
> >>>> week.
> >>>>
> >>>> We should then be ready for beta (beta == no new features, no API
> >>>> changes,
> >>>> just fixes).
> >>>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <stack@duboce.net> wrote:
> >>>>
> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> >>>>>
> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
get a
> >>>>> release out in the next week or so.
> >>>>>
> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >>>>> interested in helping out, below are the items I think we need done
> for
> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> altering
> >>>>> items, and are absent those JIRAs that are making active progress,
> >>>>> i.e. the
> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> >>>>> doing is
> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >>>>>
> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> >>>>> branch-2
> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
do it
> >>>>> between branch-1 and branch-2.
> >>>>>
> >>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
> >>>>> interfaces
> >>>>> This one is almost done. It could do with a finish, attention to
the
> >>>>> items in last comment, and then our codebase could do with another
> >>>>> sweep
> >>>>> after the spirit of this issue since a bunch has gone in since the
> pass
> >>>>> that was the basis of this issue.
> >>>>>
> >>>>> * HBASE-10504 Define Replication Interface
> >>>>> I was going to take a crack at this as part of the revamp forced
by
> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> >>>>> but if
> >>>>> anyone else is interested, be my guest.
> >>>>>
> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
> >>>>> finishing.
> >>>>>
> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
and
> >>>>> cleanup
> >>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
> >>>>> and
> >>>>> make sure symmetry.
> >>>>>
> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
be
> >>>>> done.
> >>>>>
> >>>>> Rolling upgrade will have implications for our API. Would be good
to
> >>>>> try
> >>>>> it and figure what needs fixup (as said above, according to trial
by
> >>>>> Sean,
> >>>>> we might not be too bad here):
> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> >>>>> cluster
> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
and
> >>>>> 1.x
> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >>>>>
> >>>>> * HBASE-17442 Move most of the replication related classes to
> >>>>> hbase-server package
> >>>>> The above would be good to do generally but it may make for ripples
> in
> >>>>> API so would be good to do now.
> >>>>>
> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >>>>> Balazs is working on this. The idea is that we avoid adding two
new
> >>>>> types
> >>>>> to our API, two types that are nought but curtailed, read-only views
> on
> >>>>> internals. Input if you have time appreciated.
> >>>>>
> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> hbase2
> >>>>> cluster; verify
> >>>>> Esteban is looking at this one
> >>>>>
> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >>>>> * HBASE-17143 Scan improvement
> >>>>>
> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >>>>> implementations
> >>>>> marked audience Private and returns implementations rather than
> >>>>> Interfaces.
> >>>>> In a few locations, we allow returning an alternate implementation
> >>>>> altogether which is probably something we don't want a CP doing.
To
> >>>>> that
> >>>>> end, the following issues started by Duo and Anoop need to be taken
> to
> >>>>> the
> >>>>> finish line; ideally they'd have an owner:
> >>>>>
> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
The
> >>>>> umbrella issue.
> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
and
> >>>>> RegionServerObserver
> >>>>>
> >>>>>
> >>>>> Nice-to-haves:
> >>>>>
> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> unused
> >>>>> TimeRange constructors
> >>>>>
> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> >>>>> existing in the code
> >>>>> This is the end of an old long-running project moving up on to Cell
> >>>>> Interface. We think it is done but for a few little items (deprecate
> KV
> >>>>> methods in MR and provide Cell versions instead...)
> >>>>>
> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
needs
> >>>>> fixing
> >>>>>
> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >>>>>
> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >>>>> * HBASE-14997
> >>>>> Move compareOp and Comparators out of filter to client package
> >>>>>
> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >>>>>
> >>>>> What about:
> >>>>>
> >>>>> * HBASE-18601 Remove Htrace 3.2
> >>>>> As has been noted, the HTrace API is our 'trace' API.
> >>>>>
> >>>>> If interested in any of the above and you need a legup, just ask
in
> the
> >>>>> issue and I'll be by....
> >>>>>
> >>>>> Thanks,
> >>>>> St.Ack
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net>
wrote:
> >>>>>
> >>>>> Heads-up:
> >>>>>>
> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
Theme
> is
> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
> >>>>>> netty,
> >>>>>> protobuf), purge of checked-in generated src, and
> >>>>>> master-carries-no-regions
> >>>>>> by default.
> >>>>>>
> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> will
> >>>>>> be
> >>>>>> settling the APIs and compatibility (At first blush, we are
not
> >>>>>> looking too
> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
client
> >>>>>> running
> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> >>>>>> should be
> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >>>>>> Implementations, and
> >>>>>> our shutdown of CPs accessing classes in hbase marked
> >>>>>> InterfaceAudience).
> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
guava
> >>>>>> 0.12
> >>>>>> Service showing through in our replication API and protobufs
in
> Admin
> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach
to make
> >>>>>> sure
> >>>>>> our downstreamers are up on what is coming down the pipe.
> >>>>>>
> >>>>>> Beta1 in mid-September?
> >>>>>>
> >>>>>> I encourage you to check out the items marked for hbase2:
> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> Edit
> >>>>>> as
> >>>>>> you see appropriate. Punt if you know the JIRA will not get
any
> >>>>>> attention
> >>>>>> in next month or so.
> >>>>>>
> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave
them as
> is
> >>>>>> another while but I'll boot them soon.
> >>>>>>
> >>>>>> While I have your attention:
> >>>>>>
> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> thrift
> >>>>>> to 0.10.0 will break existing clients. The change is easy enough
if
> >>>>>> folks
> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> >>>>>> (0.98?).
> >>>>>>
> >>>>>> St.Ack
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <stack@duboce.net>
wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >>>>>>>>
> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<elserj@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> ...
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I like the idea of this also hitting 2.0 as
it would make the
> >>>>>>>>>> feature a
> >>>>>>>>>> bit more "real", but am obviously a little nervous
(I have no
> >>>>>>>>>> reason
> >>>>>>>>>> to be
> >>>>>>>>>> nervous though). I am pretty happy with the
feature in terms of
> >>>>>>>>>> how
> >>>>>>>>>> much it
> >>>>>>>>>> is covered via testing.
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sounds good to me. Whats involved? Backport?
If so, +1 Josh.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Last think on space quota says that need doc too.
See 'Space
> >>>>>>>>> Quota' in
> >>>>>>>>> here:
> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>>>>>>> Does this little section need an update Josh?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> S
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
and some
> docs
> >>>>>>>> already included for 17748.  Happy to put that on my
plate if
> >>>>>>>> you're good
> >>>>>>>> with it. I can reasonably assume that no one is against
it :)
> >>>>>>>>
> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff
before we
> >>>>>>>> merged it in from the original feature branch. I'll
double check
> >>>>>>>> and update
> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Thanks Josh,
> >>>>>>> S
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message