hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Moving 2.0 forward
Date Tue, 24 Oct 2017 22:03:12 GMT
Chatting with my coworker Mr. Mike Drob, we were batting back and forth
what remains to be done. Surfacing our thoughts here so you all clued
in....Or if you think otherwise, please speak up.

We have ~13 issues to land:
https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
are meta-issues that are about process which leaves 11.

Duo and Zheng Hu are to merge the FilterList fixes improvements
(HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
API/semantic that we need to get out earlier rather than later.

Once the above is merged, HBASE-13346, change of Filter method names to
mention Cell instead of KeyValue can land.

HBASE-199048 needs a review (Anoop will probably do it), removing
IA.Private objects as params to MasterObserver... Hopefully this goes in
soon.

Duo is hard at work on trackers for flush and compaction for CPs
(HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?

I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
done w/ his work above.

I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
purges allowing CPs do direct calls against Regions in same Host.

Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.

Another day or two?

St.Ack



On Mon, Oct 23, 2017 at 2:52 PM, Stack <stack@duboce.net> wrote:

>
> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <elserj@apache.org> wrote:
>
>> +1
>>
>> I was trying to work on helping out on the outstanding alpha-4 stuff last
>> week -- will be continuing to try to do the same this week.
>>
>> If you need any help, Stack, or if others need reviews where I haven't
>> noticed on my own: feel free to @mention me.
>>
>>
> Thanks for the offer Josh. All items seem assigned and are being actively
> worked on. If you get a moment, reviews by you (or anyone else) helps move
> the process along.
>
> We need to merge in HBASE-18410 branch to pick up Filter improvements.
> Then HBASE-13346 can go in.
>
> You are already helping out on HBASE-18906, thanks. Looks like that will
> be addressed by other alpha-4s about to land.
>
> St.Ack
> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>
>
>
>
>
>
>
>
>
>> On 10/23/17 12:53 PM, Stack wrote:
>>
>>> (Reviving this thread)
>>>
>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>> refactor of the Coprocessor API shutting down access to internals marked
>>> InterfaceAudience.Private.
>>>
>>> The outstanding list is here:
>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>
>>> Please push in anything marked alpha-4 that belongs to you.
>>>
>>> If issue, talk out loud on this thread. If you need a review to land an
>>> item, shout on the issue and here; we'll help you out.
>>>
>>> As is, items are coming along nicely I'd say. We need to merge the filter
>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>
>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>> test
>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>> we
>>> beta-1).
>>>
>>> Thanks for time,
>>> S
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <stack@duboce.net> wrote:
>>>
>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>
>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>>>> it is plain that more time is needed (in spite of valiant effort so far
>>>> by
>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>> theme is
>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>>>> week.
>>>>
>>>> We should then be ready for beta (beta == no new features, no API
>>>> changes,
>>>> just fixes).
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <stack@duboce.net> wrote:
>>>>
>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>
>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>> release out in the next week or so.
>>>>>
>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>> interested in helping out, below are the items I think we need done for
>>>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>>>> items, and are absent those JIRAs that are making active progress,
>>>>> i.e. the
>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>> doing is
>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>
>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>> branch-2
>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>> between branch-1 and branch-2.
>>>>>
>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>> interfaces
>>>>> This one is almost done. It could do with a finish, attention to the
>>>>> items in last comment, and then our codebase could do with another
>>>>> sweep
>>>>> after the spirit of this issue since a bunch has gone in since the pass
>>>>> that was the basis of this issue.
>>>>>
>>>>> * HBASE-10504 Define Replication Interface
>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>> but if
>>>>> anyone else is interested, be my guest.
>>>>>
>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>> finishing.
>>>>>
>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>> cleanup
>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>> and
>>>>> make sure symmetry.
>>>>>
>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>> done.
>>>>>
>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>> try
>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>> Sean,
>>>>> we might not be too bad here):
>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>> cluster
>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>> 1.x
>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>
>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>> hbase-server package
>>>>> The above would be good to do generally but it may make for ripples in
>>>>> API so would be good to do now.
>>>>>
>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>> types
>>>>> to our API, two types that are nought but curtailed, read-only views
on
>>>>> internals. Input if you have time appreciated.
>>>>>
>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>>>> cluster; verify
>>>>> Esteban is looking at this one
>>>>>
>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>> * HBASE-17143 Scan improvement
>>>>>
>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>> implementations
>>>>> marked audience Private and returns implementations rather than
>>>>> Interfaces.
>>>>> In a few locations, we allow returning an alternate implementation
>>>>> altogether which is probably something we don't want a CP doing. To
>>>>> that
>>>>> end, the following issues started by Duo and Anoop need to be taken to
>>>>> the
>>>>> finish line; ideally they'd have an owner:
>>>>>
>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
The
>>>>> umbrella issue.
>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>> RegionServerObserver
>>>>>
>>>>>
>>>>> Nice-to-haves:
>>>>>
>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>>>> TimeRange constructors
>>>>>
>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>> existing in the code
>>>>> This is the end of an old long-running project moving up on to Cell
>>>>> Interface. We think it is done but for a few little items (deprecate
KV
>>>>> methods in MR and provide Cell versions instead...)
>>>>>
>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
needs
>>>>> fixing
>>>>>
>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>
>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>> * HBASE-14997
>>>>> Move compareOp and Comparators out of filter to client package
>>>>>
>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>
>>>>> What about:
>>>>>
>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>
>>>>> If interested in any of the above and you need a legup, just ask in the
>>>>> issue and I'll be by....
>>>>>
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net> wrote:
>>>>>
>>>>> Heads-up:
>>>>>>
>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
is
>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>> netty,
>>>>>> protobuf), purge of checked-in generated src, and
>>>>>> master-carries-no-regions
>>>>>> by default.
>>>>>>
>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
will
>>>>>> be
>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>> looking too
>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>> running
>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>> should be
>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>> Implementations, and
>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>> InterfaceAudience).
>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>> 0.12
>>>>>> Service showing through in our replication API and protobufs in Admin
>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>> sure
>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>
>>>>>> Beta1 in mid-September?
>>>>>>
>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
>>>>>> as
>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>> attention
>>>>>> in next month or so.
>>>>>>
>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
as is
>>>>>> another while but I'll boot them soon.
>>>>>>
>>>>>> While I have your attention:
>>>>>>
>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>>>> to 0.10.0 will break existing clients. The change is easy enough
if
>>>>>> folks
>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>> (0.98?).
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <stack@duboce.net> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>
>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<elserj@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
make the
>>>>>>>>>> feature a
>>>>>>>>>> bit more "real", but am obviously a little nervous
(I have no
>>>>>>>>>> reason
>>>>>>>>>> to be
>>>>>>>>>> nervous though). I am pretty happy with the feature
in terms of
>>>>>>>>>> how
>>>>>>>>>> much it
>>>>>>>>>> is covered via testing.
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so,
+1 Josh.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Last think on space quota says that need doc too. See
'Space
>>>>>>>>> Quota' in
>>>>>>>>> here:
>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
some docs
>>>>>>>> already included for 17748.  Happy to put that on my plate
if
>>>>>>>> you're good
>>>>>>>> with it. I can reasonably assume that no one is against it
:)
>>>>>>>>
>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
we
>>>>>>>> merged it in from the original feature branch. I'll double
check
>>>>>>>> and update
>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>
>>>>>>>>
>>>>>>> Thanks Josh,
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message