phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Istvan Toth <st...@cloudera.com.INVALID>
Subject Re: Roadmap to releasing Phoenix 5.1, PQS 6.0 and Connectors 6.0
Date Wed, 05 Aug 2020 05:11:09 GMT
Hi!

Strongly Consistent Global Indexes is indeed the killer new feature in 5.1,
and it's important to have it working as well as possible.
I was only vaguely aware of the deficiencies in master, thanks for
explaining it (and working on fixing them)

Thanks
Istvan

On Tue, Aug 4, 2020 at 9:22 PM Geoffrey Jacoby <gjacoby@apache.org> wrote:

> Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-)
> ) for all your hard work getting master and the ancillary projects into a
> releasable state.
>
> An additional task that I think should happen before 5.1 can be released is
> getting the indexing code in master up to parity with 4.x. As you may know,
> between 4.14 and the upcoming 4.16, a lot of work has been done to build a
> new, self-repairing consistent secondary index framework for Phoenix. Most
> of that work has been ported to the master branch, but there are still some
> significant gaps.
>
> The biggest gap comes from the new indexing framework relying heavily on
> Phoenix's SCN "lookback" feature to do point-in-time selects during index
> creation and verification. SCN has in the past been an unreliable tool,
> because HBase's flush and major compaction code cleans up expired versions
> and removes chunks of prior history. To work around this, in 4.16 we've
> introduced "max lookback age" in PHOENIX-5645, which allows operators to
> configure a moving window during which compactions and flushes will not
> purge versions for any history.
>
> PHOENIX-5645 doesn't exist in master, because the coprocessor changes in
> HBase 2.0 made implementing the needed compaction hooks impossible.
> HBASE-24321, released in HBase 2.3, makes it possible again, though only
> for Phoenix builds using the 2.3 profile. (Since it adds to an interface,
> HBase compatibility guidelines prevent HBASE-24321 from being backported to
> 2.1 or 2.2.)
>
> So, for 5.1 I believe we need forward ports for :
> PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER
> PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies
> before or after max lookback age) - BLOCKER
> PHOENIX-5928 (Simplification / perf improvement for index builds)
> PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER
> PHOENIX-5951 (Configurable failure logging for past-max-lookback rows)
> PHOENIX-6058 (Better behavior on verification when max lookback is disabled
> -- needed for HBase 2.1 and 2.2 profiles) - BLOCKER
>
> Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-)
>  Happy to discuss what is and isn't a blocker.
>
> I plan to do PHOENIX-5881 next week, and the rest depends on it and will
> follow afterward.
>
> Thanks,
>
> Geoffrey
>
>
> On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <stoty@apache.org> wrote:
>
> > Hi!
> >
> > It's been more than two years since we've released 5.0.0, and almost as
> > long since Connectors and PQS have been split from the main repo.
> >
> > I believe that we are now at the point where we've solved, or are close
> to
> > solving the issues that have prevented us from releasing a useful and
> > relevant 5.1.0 , as well as making an actual releases of PQS and
> Connectors
> > that are usable with both 5.x and 4.x.
> >
> > The two major blockers that are still open are
> >
> >    - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it
> >    - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master
> > branch
> >
> > but I hope that we can wrap those up in the next few weeks.
> >
> > This is going to be a complex process, as we'll have to release new
> > versions of ALL of our components. To recap, the affected projects, (and
> > their dependencies):
> >
> >    - phoenix-thirdparty
> >    - tephra
> >    - omid (phoenix-thirdparty)
> >    - phoenix (tehpra ?, omid, phoenix-thirdparty)
> >    - PQS
> >    - Connectors
> >
> > The 5.1 release is also a point where we can revisit the decision to
> > support Tephra. We have inherited those projects because of low developer
> > interest, and it hasn't increased visibly since we've adopted them.
> > Rajeshbabu and Josh have done some analysis and, as a part of our day
> job,
> > are investing time first with Omid to ensure it's functional with the
> rest
> > of Phoenix in its new home/packaging.
> >
> > Tephra also carries the technical debt of being dependent on the
> > discontinued Twill library, which in turn is locked to oid Guava
> versions.
> > In TEPHRA-308 I am implementing the stopgap solution of shading both
> away,
> > so it is not a blocker for 5.1, but concentrating on one library would
> > probably be a smarter use of the almost non-existent developer time that
> > goes into maintaining our transactional solution.
> >
> > I plan to add a profile to build Phoenix without Tephra, thus avoiding
> the
> > problematic dependencies that it has. (Alternatively, the default can be
> > omitting Tephra, and defining a profile where it is added.)
> >
> >
> > The effect on 4.x
> >
> > Short-term, the above releases do not affect 4.x, as it can stay on the
> old
> > omid and tephra dependencies. Having an official release of PQS and
> > Connectors is a clear win, and Richard Antal is also working on updating
> > some of the connectors for 4.x.
> >
> > Mid-term, updating to the new Omid version will bring in the
> > phoenix-thirdparty dependency to 4.x, and I think it would be smart to
> > backport the phoenix-thirdparty changes to 4.x as well.
> >
> > I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
> > releases that are close feature and bugfix wise would be advantageous in
> > terms of documenting and communicating them to the users, but it hasn't
> > stopped either branch from releasing in the past, so releasing 5.1 and
> 4.16
> > close together would be a nice-to-have, but not a show stopper.
> >
> > Also, I have concentrated on the build and dependencies side of Phoenix.
> > AFAICT there are no major new features going in now that would warrant
> > delaying the release, I can mostly see the usual fixes and optimizations
> > these days among the commits.
> >
> > Thanks for taking the time to read this.
> >
> > Looking forward to your questions, comments, and opinion.
> >
> > regards
> >
> > Istvan
> >
>


-- 
*István Tóth* | Staff Software Engineer
stoty@cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
<https://www.cloudera.com/>
------------------------------

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message