Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CAAAYAnP79BvM9NYnkuxcXOtRN7jmw6jbMi2noL=5Us3nD6OVtw@mail.gmail.com>
References: <64DC0B0F-E4F4-4057-B6D4-99F1AB870D19@gmail.com>
 <CA+RK=_D_8SiyqVT9RJt6zjOwdxU2TeRoKUzeRJGwBBRcNH2j=w@mail.gmail.com>
 <CADcMMgEdJvxKSR3wvru-Q+JQZfni3jK5t6DuBVj1GFn6F=fksg@mail.gmail.com>
 <CADcMMgE0_ASBM7JZcgPDLdhU6kCWVeD5JUKpjh4wrb-6xKd=xg@mail.gmail.com>
 <CAAAYAnO1uunOUuBu7PZ8=1dqF0oj8yM=e1CA-wgSXWyVQL5CVw@mail.gmail.com>
 <CADcMMgG6-fo5O13KdKk23fO4XWwttzg0c4WSV6NxUQP=4CYfQA@mail.gmail.com>
 <CADcMMgHvZDvPjfVBQOH9b0g8=0wCAfejMnM7Eut0gvnEKrwfew@mail.gmail.com>
 <5627f4ff-203e-8ef9-4915-68f63a8c18e5@apache.org> <CADcMMgHq_FZzbZ0i1-JO=8G-whTFUZ+R8GPn7wfb5-am=6Qx-Q@mail.gmail.com>
 <090b5156-0850-07e7-0c33-c252a917476b@apache.org> <CADcMMgHhuzyboizYiOV7r7dO+ZVbuzBwS2Z8vPZAekSkJ18W9w@mail.gmail.com>
 <CADcMMgGGrvGcoCiLqQ5kO_TEcDJ7YV=B26i6f39OqJMA6X9-MQ@mail.gmail.com>
 <CADcMMgEm+aJDLz2jx6FQVR5Kq4dVZKfer45Um895Rkzu9Y3kJw@mail.gmail.com>
 <CADcMMgFf9_e1vuSZYTGP6uA93DEVH086q+7OnTfzjxqufaxVvw@mail.gmail.com>
 <CADcMMgEPioyM+xQMhGSgjd5LBY478rtEHMeV7eb8Parun7cn5w@mail.gmail.com>
 <0988dc33-b7ee-42c3-d2b0-e3834d969f3c@apache.org> <CADcMMgGgJ-msZve6SOsVGy_So1g0WseSukuK+ecEyDzALHkh+w@mail.gmail.com>
 <CADcMMgF4o9oq3bzqXG3az+3+WZFcxZ=fZ5jm7ZtCODJi=1yMHg@mail.gmail.com> <CAAAYAnP79BvM9NYnkuxcXOtRN7jmw6jbMi2noL=5Us3nD6OVtw@mail.gmail.com>
From: =?UTF-8?B?5byg6ZOOKER1byBaaGFuZyk=?= <palomino219@gmail.com>
Date: Thu, 26 Oct 2017 09:51:06 +0800
Message-ID: <CAAAYAnNYGUUa-mCkkjpUoLKNiGi+H7pWWu4b6f+VxHQyeW6S0Q@mail.gmail.com>
Subject: Re: Moving 2.0 forward
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary="001a11482214dbc839055c696765"
archived-at: Thu, 26 Oct 2017 01:51:17 -0000

--001a11482214dbc839055c696765
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

OK, skimmed,  we are in trouble! The in memory compaction just use the same
constructor with normal compaction to construct a StoreScanner, and use it
to do compaction...

We have to provide several preXXX and postXXX for it, at least we should
allow user reset TTL and max versions, and also do filtering on the scanner=
.

Thanks.


2017-10-26 9:41 GMT+08:00 =E5=BC=A0=E9=93=8E(Duo Zhang) <palomino219@gmail.=
com>:

> When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> HBASE-19033, I found that there maybe a hole in our CP hools. It is the i=
n
> memory compaction.
>
> As Anoop suggested in HBASE-19001, we still need to give user the ability
> to extend the max versions and TTL config, so I plan to add back the hook=
s
> above to let CP users can change the max versions and TTL of a ScanInfo
> object. But I'm not sure whether in memory compaction will also discard
> expired cells, if so then we are in trouble...
>
> Thanks.
>
> 2017-10-25 6:03 GMT+08:00 Stack <stack@duboce.net>:
>
>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>> what remains to be done. Surfacing our thoughts here so you all clued
>> in....Or if you think otherwise, please speak up.
>>
>> We have ~13 issues to land:
>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About tw=
o
>> are meta-issues that are about process which leaves 11.
>>
>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some change=
d
>> API/semantic that we need to get out earlier rather than later.
>>
>> Once the above is merged, HBASE-13346, change of Filter method names to
>> mention Cell instead of KeyValue can land.
>>
>> HBASE-199048 needs a review (Anoop will probably do it), removing
>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>> soon.
>>
>> Duo is hard at work on trackers for flush and compaction for CPs
>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>
>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>> is
>> done w/ his work above.
>>
>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>> the
>> purges allowing CPs do direct calls against Regions in same Host.
>>
>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>
>> Another day or two?
>>
>> St.Ack
>>
>>
>>
>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <stack@duboce.net> wrote:
>>
>> >
>> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <elserj@apache.org> wrote=
:
>> >
>> >> +1
>> >>
>> >> I was trying to work on helping out on the outstanding alpha-4 stuff
>> last
>> >> week -- will be continuing to try to do the same this week.
>> >>
>> >> If you need any help, Stack, or if others need reviews where I haven'=
t
>> >> noticed on my own: feel free to @mention me.
>> >>
>> >>
>> > Thanks for the offer Josh. All items seem assigned and are being
>> actively
>> > worked on. If you get a moment, reviews by you (or anyone else) helps
>> move
>> > the process along.
>> >
>> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
>> > Then HBASE-13346 can go in.
>> >
>> > You are already helping out on HBASE-18906, thanks. Looks like that wi=
ll
>> > be addressed by other alpha-4s about to land.
>> >
>> > St.Ack
>> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >> On 10/23/17 12:53 PM, Stack wrote:
>> >>
>> >>> (Reviving this thread)
>> >>>
>> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>> >>> refactor of the Coprocessor API shutting down access to internals
>> marked
>> >>> InterfaceAudience.Private.
>> >>>
>> >>> The outstanding list is here:
>> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >>>
>> >>> Please push in anything marked alpha-4 that belongs to you.
>> >>>
>> >>> If issue, talk out loud on this thread. If you need a review to land
>> an
>> >>> item, shout on the issue and here; we'll help you out.
>> >>>
>> >>> As is, items are coming along nicely I'd say. We need to merge the
>> filter
>> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>> >>>
>> >>> Post alpha-4, we'll have to hunt down our downstreamers and help the=
m
>> >>> test
>> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
>> >>> downstreamers know what to expect (or we discover what we missed
>> BEFORE
>> >>> we
>> >>> beta-1).
>> >>>
>> >>> Thanks for time,
>> >>> S
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <stack@duboce.net> wrote:
>> >>>
>> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should =
be
>> >>>> time, if we all sprint, for the public-facing API fixes to be done.
>> >>>>
>> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha=
3
>> but
>> >>>> it is plain that more time is needed (in spite of valiant effort so
>> far
>> >>>> by
>> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>> >>>> theme is
>> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>> following
>> >>>> week.
>> >>>>
>> >>>> We should then be ready for beta (beta =3D=3D no new features, no A=
PI
>> >>>> changes,
>> >>>> just fixes).
>> >>>>
>> >>>> Thanks,
>> >>>> St.Ack
>> >>>>
>> >>>>
>> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <stack@duboce.net> wrote:
>> >>>>
>> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on i=
t.
>> >>>>>
>> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to ge=
t
>> a
>> >>>>> release out in the next week or so.
>> >>>>>
>> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>> >>>>> interested in helping out, below are the items I think we need don=
e
>> for
>> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
>> altering
>> >>>>> items, and are absent those JIRAs that are making active progress,
>> >>>>> i.e. the
>> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>> >>>>> doing is
>> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
>> >>>>>
>> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>> >>>>> branch-2
>> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
>> it
>> >>>>> between branch-1 and branch-2.
>> >>>>>
>> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
>> Private
>> >>>>> interfaces
>> >>>>> This one is almost done. It could do with a finish, attention to t=
he
>> >>>>> items in last comment, and then our codebase could do with another
>> >>>>> sweep
>> >>>>> after the spirit of this issue since a bunch has gone in since the
>> pass
>> >>>>> that was the basis of this issue.
>> >>>>>
>> >>>>> * HBASE-10504 Define Replication Interface
>> >>>>> I was going to take a crack at this as part of the revamp forced b=
y
>> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service=
'
>> >>>>> but if
>> >>>>> anyone else is interested, be my guest.
>> >>>>>
>> >>>>> * HBASE-14996 Some more API cleanup for 2.0
>> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>> >>>>> finishing.
>> >>>>>
>> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>> and
>> >>>>> cleanup
>> >>>>> Needs a pass. Small issue I think. Could also look at new
>> AsyncClient
>> >>>>> and
>> >>>>> make sure symmetry.
>> >>>>>
>> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to =
be
>> >>>>> done.
>> >>>>>
>> >>>>> Rolling upgrade will have implications for our API. Would be good =
to
>> >>>>> try
>> >>>>> it and figure what needs fixup (as said above, according to trial =
by
>> >>>>> Sean,
>> >>>>> we might not be too bad here):
>> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>> >>>>> cluster
>> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master a=
nd
>> >>>>> 1.x
>> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>> >>>>>
>> >>>>> * HBASE-17442 Move most of the replication related classes to
>> >>>>> hbase-server package
>> >>>>> The above would be good to do generally but it may make for ripple=
s
>> in
>> >>>>> API so would be good to do now.
>> >>>>>
>> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>> >>>>> Balazs is working on this. The idea is that we avoid adding two ne=
w
>> >>>>> types
>> >>>>> to our API, two types that are nought but curtailed, read-only
>> views on
>> >>>>> internals. Input if you have time appreciated.
>> >>>>>
>> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>> hbase2
>> >>>>> cluster; verify
>> >>>>> Esteban is looking at this one
>> >>>>>
>> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>> >>>>> * HBASE-17143 Scan improvement
>> >>>>>
>> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
>> >>>>> implementations
>> >>>>> marked audience Private and returns implementations rather than
>> >>>>> Interfaces.
>> >>>>> In a few locations, we allow returning an alternate implementation
>> >>>>> altogether which is probably something we don't want a CP doing. T=
o
>> >>>>> that
>> >>>>> end, the following issues started by Duo and Anoop need to be take=
n
>> to
>> >>>>> the
>> >>>>> finish line; ideally they'd have an owner:
>> >>>>>
>> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
=3D
>> The
>> >>>>> umbrella issue.
>> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver a=
nd
>> >>>>> RegionServerObserver
>> >>>>>
>> >>>>>
>> >>>>> Nice-to-haves:
>> >>>>>
>> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>> unused
>> >>>>> TimeRange constructors
>> >>>>>
>> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>> >>>>> existing in the code
>> >>>>> This is the end of an old long-running project moving up on to Cel=
l
>> >>>>> Interface. We think it is done but for a few little items
>> (deprecate KV
>> >>>>> methods in MR and provide Cell versions instead...)
>> >>>>>
>> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>> needs
>> >>>>> fixing
>> >>>>>
>> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
>> >>>>>
>> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
>> >>>>> * HBASE-14997
>> >>>>> Move compareOp and Comparators out of filter to client package
>> >>>>>
>> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
>> >>>>>
>> >>>>> What about:
>> >>>>>
>> >>>>> * HBASE-18601 Remove Htrace 3.2
>> >>>>> As has been noted, the HTrace API is our 'trace' API.
>> >>>>>
>> >>>>> If interested in any of the above and you need a legup, just ask i=
n
>> the
>> >>>>> issue and I'll be by....
>> >>>>>
>> >>>>> Thanks,
>> >>>>> St.Ack
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net> wrote:
>> >>>>>
>> >>>>> Heads-up:
>> >>>>>>
>> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Them=
e
>> is
>> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
>> >>>>>> netty,
>> >>>>>> protobuf), purge of checked-in generated src, and
>> >>>>>> master-carries-no-regions
>> >>>>>> by default.
>> >>>>>>
>> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>> will
>> >>>>>> be
>> >>>>>> settling the APIs and compatibility (At first blush, we are not
>> >>>>>> looking too
>> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1 clien=
t
>> >>>>>> running
>> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>> >>>>>> should be
>> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
>> >>>>>> Implementations, and
>> >>>>>> our shutdown of CPs accessing classes in hbase marked
>> >>>>>> InterfaceAudience).
>> >>>>>> We'll also have purged thirdparty classes from our API; e.g. guav=
a
>> >>>>>> 0.12
>> >>>>>> Service showing through in our replication API and protobufs in
>> Admin
>> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>> make
>> >>>>>> sure
>> >>>>>> our downstreamers are up on what is coming down the pipe.
>> >>>>>>
>> >>>>>> Beta1 in mid-September?
>> >>>>>>
>> >>>>>> I encourage you to check out the items marked for hbase2:
>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>> Edit
>> >>>>>> as
>> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
>> >>>>>> attention
>> >>>>>> in next month or so.
>> >>>>>>
>> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>> as is
>> >>>>>> another while but I'll boot them soon.
>> >>>>>>
>> >>>>>> While I have your attention:
>> >>>>>>
>> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>> thrift
>> >>>>>> to 0.10.0 will break existing clients. The change is easy enough =
if
>> >>>>>> folks
>> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>> >>>>>> (0.98?).
>> >>>>>>
>> >>>>>> St.Ack
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <stack@duboce.net> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>> >>>>>>>>
>> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<elserj@apache.org>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> ...
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>> >>>>>>>>>> feature a
>> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>> >>>>>>>>>> reason
>> >>>>>>>>>> to be
>> >>>>>>>>>> nervous though). I am pretty happy with the feature in terms =
of
>> >>>>>>>>>> how
>> >>>>>>>>>> much it
>> >>>>>>>>>> is covered via testing.
>> >>>>>>>>>>
>> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
>> >>>>>>>>> Quota' in
>> >>>>>>>>> here:
>> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=3Dh.wuw3a6jukzo5
>> >>>>>>>>> Does this little section need an update Josh?
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> S
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>> docs
>> >>>>>>>> already included for 17748.  Happy to put that on my plate if
>> >>>>>>>> you're good
>> >>>>>>>> with it. I can reasonably assume that no one is against it :)
>> >>>>>>>>
>> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before w=
e
>> >>>>>>>> merged it in from the original feature branch. I'll double chec=
k
>> >>>>>>>> and update
>> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>> Thanks Josh,
>> >>>>>>> S
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >
>>
>
>

--001a11482214dbc839055c696765--