From dev-return-59652-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Tue Jan 14 18:12:18 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9C3D618061A for ; Tue, 14 Jan 2020 19:12:17 +0100 (CET) Received: (qmail 76573 invoked by uid 500); 14 Jan 2020 18:12:16 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 76454 invoked by uid 99); 14 Jan 2020 18:12:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jan 2020 18:12:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BEACFC0870 for ; Tue, 14 Jan 2020 18:12:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.674 X-Spam-Level: * X-Spam-Status: No, score=1.674 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id q-RfIwbdkKfI for ; Tue, 14 Jan 2020 18:12:10 +0000 (UTC) Received-SPF: Softfail (mailfrom) identity=mailfrom; client-ip=74.6.129.81; helo=sonic317-26.consmr.mail.bf2.yahoo.com; envelope-from=larsh@apache.org; receiver= Received: from sonic317-26.consmr.mail.bf2.yahoo.com (sonic317-26.consmr.mail.bf2.yahoo.com [74.6.129.81]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id DE2F5BC58D for ; Tue, 14 Jan 2020 18:12:09 +0000 (UTC) X-ASF-DKIM-Sig: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1579025523; bh=3AYLjO/8dPtSAu24gi3dHwfP+X6YD2TKqB+2bEDjdP0=; h=Date:From:To:In-Reply-To:References:Subject:From:Subject; b=Hi1D1lQdN/ZWxeDxM0B/5Ggd2CnQambaXbRzGygjVJPWrORrxDOU7JaKrHEu87sAuyUiltTFDu+9WSp0Zlv2Ik4BT55Agn6XYWTjQgoNKj0+pjrsiPlpi18941/zLs0ZzPXUaHyLMQMXGrMWkf1SGjdCyFjg09alUIX3RTPX/WQZrC6ybgQ2pHr1CXIOC6zuIO8W7+katfYDeE4+guDDG9I4UzSX5h4VlllwDFtTqMHLYGy8ltk0Kf/vEwvLZLB/I5nRTnrvAOCnCYluIJVEaQ/dn1fUwZGA0ac3Eq8Mhi2SgiDdXRVpKlp19xAldHcaZ31WPV8DIHCOcqSZAB4qoQ== X-YMail-OSG: .nxLkk0VM1mnWcIk_4zPl79FwuoTcZxALWyfRstXRpHy9k2Sb3rqj3E9ZpypJ9y FKDsGDu3s9TYK7KjxyZpW2mxva0PdoC3mKD_TORkU1NutCB2yp7FEIFlYvJcfCnU761XxpDNpDgN VyeUITvN9ago8aX0qVbJB4wQ5cQZCw6Z.My1Cz1pbdnCwV5iTl65i7RzNMJkV9KvdlyytVvoFAEO 5O1fS7iAiouuTxNpJExlfIZpMdB.xlnza8pSTf1.gYebNz6j6AyeU4q911O9fYhncPrw97c9GkH8 6Hkf.E.F09VP1f3eNd2HzZH5Grkh2w54vZJtUep2VhsD6TVV.gDPSCRq2ZAFcWPaItVU98bJsdb_ pUuoVZXrGvV2x0RFKN36mRtthkbnXllrP4yg8Z_yK5EYDKvEBQwviY6p1ribKulNF80llv7zLxZl _bgv97sb.kCFKQ1n3i3mOPYofhwNeAUI.fdIQ9XMHxN63C01ts1C8lLm5J3TDjutdU4FeyS1V1P3 okWHOZJd0e5I8pdGZlnV.Yt5tQZlU8dmbvvO8E3w1Jsxnbzio1Vf9eCfsFpGO2i2oO1DFJS2eNkt ZyrkFsKMbIkz52aM57HE0YH2pr3XBsQEYOYVqvEk5VD0Wr.s66Q.1iMC5CevhN5BYGnLR_F_jwNy I2j0gGHlh3wi3QZ4T8Y9.AgqE170lMusrTMQKmz2ZPjkf5Z8oNqL3moX9Mvkpm0GYuveMmejh59f hfPSloY1JXcnmZW5LUk6q_RZvYigwSaJZrz8HicMxKhL1.G2195S75VMozKtVgbWk45nxy5L8HPk gKLPqpb1JefBY7uGfG1speIUJCMZu_1sqDueEUu05Jgvy_wkBaltQYz6qEJbhHVawT4Fc5ySUMF9 HtHjrMrJ7lwYf49N93Gwr3fZtb9Hi7OHnuwyp2rfVRm9s057LCrkOoLc1FNqjNUsaCGK_Iq9xyht ..XRl2F8boctAyJcAHRsD2J7AIRfCnSXWjHfQ7AtuCSRzvHPaIrCzBc3xi9uMrdHRfaIWiyepREb bRvvBDUpBlRmPPzaoqv7nUiJambIEC9uARQyLi7TThmLOHoNlCOM20dJnfUMHdpZqytdjSfSSEnX lyhKHKKWY5JUXMzFse8VP5NrPW6dm4p0.KAHu0dC82hqhmmZjyZeafdf_29P_DAwoHK4672hZHiW COFIsuitlZDNccRCg.iXmzrExcMQJVIeDOSaWNWw6k1H.QF.1qiIZywUWyDKzzibBD2Z5p7JIcsA Zkp_PoO4hpEFj_98WLPQ1bIg7Yk_oIgtk6afeQw4XHWLmkX6MiwzGyHg4bJD7IEn2s_jAX34c5p1 7J8SNiRtJWx97fSiTvQ-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.bf2.yahoo.com with HTTP; Tue, 14 Jan 2020 18:12:03 +0000 Date: Tue, 14 Jan 2020 18:12:02 +0000 (UTC) From: "larsh@apache.org" To: dev@phoenix.apache.org Message-ID: <1832420955.7579891.1579025522291@mail.yahoo.com> In-Reply-To: References: <1e5d75e2-8fec-529e-1a2f-04380dc6e2ee@apache.org> <1517421907.550472.1576686560181@mail.yahoo.com> <559c0443-75d6-0ff6-a2bf-997ac36163d7@apache.org> <1318836778.946468.1576781683455@mail.yahoo.com> <1885416014.1127290.1576836262807@mail.yahoo.com> <11e4ce3e-4b5d-639e-9e65-605e9f558a4d@apache.org> Subject: Re: Moving Phoenix master to Hbase 2.2 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_7579890_385419341.1579025522288" X-Mailer: WebService/1.1.14873 YMailNorrin Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36 ------=_Part_7579890_385419341.1579025522288 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Does somebody volunteer to take this up? I can see whether I can a resource where I work, but it's highly uncertain. It would need a bit of digging and design work to see how we would abstract= the HBase interface in the most effective way. As mentioned below, Tephra did a good job at this and could serve as an exa= mple here. (Not dinging OMID, OMID does most of it's work client side and d= oesn't need these abstractions.) -- Lars On Tuesday, January 14, 2020, 01:13:36 AM PST, Istv=C3=A1n T=C3=B3th wrote: =20 =20 Yes, the HBase API signatures change between versions, so we need to compile each compat module against a specific HBase. Whether I can define an internal compatibility API that is switchable at run (startup) time without a performance hit remains to be seen. Istv=C3=A1n On Tue, Jan 14, 2020 at 3:21 AM Josh Elser wrote: > Agree that trying to wrangle branches is just too frustrating and > error-prone. > > It would also be great if we could have a single Phoenix jar that works > across HBase versions, but would not die on that hill :) > > On 12/20/19 5:04 AM, larsh@apache.org wrote: > >=C2=A0 I said _provided_ they can be isolated easily :) (I meant it in t= he > sense of assuming it's easy). > > As I said though, Tephra has a similar problem and they did a really > good job isolating HBase versions. We can learn from them. Sometimes they > isolate the change only, and sometimes the class needs to be copied, but > even then it's the one class that is copied, not another branch that need= s > to be kept in sync. > > > > This may also drive the desperately necessary refactoring of Phoenix to > make these things easier to isolate, or to reduce the copying to a minimu= m. > And we'd need to think through testing carefully. > > > > The branch per Phoenix and HBase version is too complex, IMHO. And the > complex branch to HBase version mapping that Istvan outlines below confir= ms > that. > > > > We should all take a brief look at the Tephra solution and see whether > we can apply that. (And since Tephra is part of the fold now, perhaps > someone can help there...?) > > Cheers. > > -- Lars > > > >=C2=A0 =C2=A0 =C2=A0 On Thursday, December 19, 2019, 8:34:15 PM GMT+1, G= eoffrey Jacoby < > gjacoby@gmail.com> wrote: > > > >=C2=A0 Lars, > > > > I'm curious why you say the differences are easily isolated -- many of > the > > core classes of Phoenix either directly inherit HBase classes or > implement > > HBase interfaces, and those can vary between minor versions. (See my > above > > example of a new coprocessor hook on BaseRegionObserver.) > > > > Geoffrey > > > > On Thu, Dec 19, 2019 at 10:54 AM larsh@apache.org > wrote: > > > >>=C2=A0 =C2=A0 Yep. The differences are pretty minimal - provided they c= an be > isolated > >> easily. > >> Tephra might be a pretty good model. It supports various versions of > HBase > >> in a single branch and has similar issues as Phoenix (coprocessors, > etc). > >> -- Lars > >>=C2=A0 =C2=A0 =C2=A0 On Thursday, December 19, 2019, 7:07:51 PM GMT+1, = Josh Elser < > >> elserj@apache.org> wrote: > >> > >>=C2=A0 =C2=A0 To clarify, you think that compat modules are better than= that > >> separate-branches model in 4.x? > >> > >> On 12/18/19 11:29 AM, larsh@apache.org wrote: > >>> This is really hard to follow. > >>> > >>> I think we should do the same with HBase dependencies in Phoenix that > >> HBase does with Hadoop dependencies. > >>> > >>> That is:=C2=A0 We could have a maven module with the specific HBase v= ersion > >> dependent code. > >>> Btw. Tephra does the same... A module for HBase version specific code= . > >>> -- Lars > >>> > >>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 On Tuesday, December 17, 2019, 10:00:31 AM= GMT+1, Istvan Toth < > >> stoty@apache.org> wrote: > >>> > >>>=C2=A0 =C2=A0 What do you think about tying the minor releases to Hbas= e minor > releases > >>> (not necessarily one-to-one) > >>> > >>> for example (provided 5.1 is 2020H1) > >>> > >>> 5.0.0 -> HB 2.0 > >>> 5.1.0 -> HB 2.2.2 (and whatever 2.1 is API compatible with it) > >>> 5.1.x -> HB 2.2.x (treat as maintenance branch, no major new features= ) > >>> 5.2.0 -> HB 2.3.0 (if released by that time) > >>> 5.2.x -> HB 2.3.x (treat as maintenance branch, no major new features= ) > >>> 5.3.0 -> HB 2.3.x (if there is no new major/minor Hbase release) > >>> master -> latest released HBase version > >>> > >>> Alternatively, we could stick with the same HBase version for patch > >>> releases that we used for the first minor release. > >>> > >>> This would limit the number of branches that we have to maintain in > >>> parallel, while providing maintenance branches for older releases, an= d > >>> timely-ish Phoenix releases. > >>> > >>> The drawback is that users of old HBase versions won't get the latest > >>> features, on the other hand they can expect more polish. > >>> > >>> Istvan > >>> > >>> On Thu, Dec 12, 2019 at 8:05 PM Geoffrey Jacoby > >> wrote: > >>> > >>>> Since HBase 2.0 is EOM'ed, I'm +1 for not worrying about 2.0.x > >>>> compatibility with the 5.x branch going forward. > >>>> > >>>> Given how coupled Phoenix is to the implementation details of HBase > >> though, > >>>> I'm not sure trying to abstract those away to keep one Phoenix branc= h > >> per > >>>> HBase major version is practical, however. At the least, it would be > >> really > >>>> complex. > >>>> > >>>> For example, in the new year I plan to return to working on the chan= ge > >> data > >>>> capture and Phoenix-level replication features, both of which depend > on > >>>> WALKey interface changes and a new RegionObserver coprocessor hook > >>>> introduced in HBASE-22622 and HBASE-22623. This was released in HBas= e > >> 1.5 > >>>> and will be in the forthcoming HBase 2.3. While the HBase community = is > >>>> discussing EOMing 1.3 right now, and maybe 1.4 will go in the medium > >> term, > >>>> I don't see all pre-2.3 branch-2's getting deprecated anytime soon. > >>>> > >>>> So there will be at least two significant features that can only exi= st > >> in > >>>> some but not all of our 4.x and 5.x branches. > >>>> > >>>> Geoffrey > >>>> > >>>> On Thu, Dec 12, 2019 at 8:21 AM Josh Elser wrote= : > >>>> > >>>>> As much as possible, I'd like to avoid us getting into another > >> situation > >>>>> with 5.x where we have multiple branches. My hope was/is that we ca= n > >>>>> keep one Phoenix5 branch that works against an acceptable set of > HBase > >>>>> branches. > >>>>> > >>>>> To me, that acceptable set of HBase branches is _a_ 2.1 and 2.2 > >> release. > >>>>> I don't think we need to support all 2.1.x or 2.2.x, nor do I think > we > >>>>> need to keep trying to maintain 2.0.x as it's already end of suppor= t > by > >>>>> the HBase community. > >>>>> > >>>>> Thanks for updating your PR. I'll add this to my review queue. > >>>>> > >>>>> On 12/12/19 1:52 AM, Istvan Toth wrote: > >>>>>> Hi! > >>>>>> > >>>>>> I'd like to start a conversation about supporting HBase 2.2. in th= e > >>>>>> master branch. > >>>>>> > >>>>>> https://issues.apache.org/jira/browse/PHOENIX-5268 has a slightly > out > >>>> of > >>>>>> date, but functional PR for HBase 2.2 support on master. (Please > >> review > >>>>>> and comment if you have the time, I'll try to update the PR in the > >> next > >>>>>> few days) > >>>>>> > >>>>>> The reason that it is not a straightforward decision to merge it i= s > >>>> that > >>>>>> applying that patch breaks compatibility with HBase 2.0.1, the > current > >>>>>> base. > >>>>>> > >>>>>> I can see the following outcomes: > >>>>>> > >>>>>> - Do nothing > >>>>>> - Move master to HBase 2.2.2 > >>>>>> - Fork master to Hbase-2.0 and Hbase-2.2 branches > >>>>>> - Build time compatibility modules > >>>>>> - Run time compatibility modules > >>>>>> - Something that I haven't thought of > >>>>>> > >>>>>> > >>>>>> Doing nothing is obviously not a long term solution, as the curren= t > >>>>>> master doesn't work with any of the currently supported HBase > >> branches, > >>>>>> but we may postpone the inevitable. > >>>>>> > >>>>>> Simply moving master to HBase 2.2 is the most attractive solution > from > >>>> a > >>>>>> pure developer POV, but there may be other considerations. > >>>>>> > >>>>>> Having multiple masters for 2.0 and 2.2 is simple from a code > >>>>>> perspective, but maintaining two branches is a non-trivial amount = of > >>>>>> additional work. (See the 4.x situation) > >>>>>> > >>>>>> Moving the HBase version dependent stuff into a separate module, a= nd > >>>>>> choosing at build time is not pretty from a code POV, but saves us > the > >>>>>> hassle of maintaining multiple branches, while maintaining > >>>> compatibility > >>>>>> with multiple=C2=A0 HBase versions, and can handle future API chan= ges as > >>>> well > >>>>>> from a single branch. Doing something like this could have saved u= s > >> the > >>>>>> effort of maintaining three separate 4.x branches. > >>>>>> > >>>>>> I feel that since Phoenix is closely timed to HBase, and requires > >>>>>> cluster-wide HBase configuration to work anyway, handling the > >> different > >>>>>> HBase versions from the same binary/JAR is not worth the effort. > >>>>>> > >>>>>> Please share your thoughts! > >>>>>> > >>>>>> regards > >>>>>> Istvan > >>>>>> > >>>>> > >>>> > >>> > >>> > >> > > > > > --=20 *Istv=C3=A1n T=C3=B3th* | Sr. Software Engineer t. (36) 70 283-1788 stoty@cloudera.com [image: Cloudera] [image: Cloudera on Twitter] [image: Cloudera on Facebook] [image: Cloudera on LinkedIn] ------------------------------ =20 ------=_Part_7579890_385419341.1579025522288--