hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject HAWQ integration to Apache bigdata stack: remaining steps
Date Mon, 25 Apr 2016 19:56:50 GMT
guys,

I wanted to put together a list of remaining steps needed before we can
declare Hawq to be a good citizen of Apache Bigtop (aka Apache bigdata stack).

I have put together a JIRA [1] to track these points, and here's the gist of
it for the reader's convenience. Please ping me if you have any questions or
follow up questions.

Regards,
  Cos

The overview of the remaining steps and the overall status of the integration work.

*External dependencies*
- the biggest issue was and remains the use of libthrift, which isn't packaged,
provided nor supported by anyone. Right now, Bigtop-HAWQ integration branch
[uses|https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob_plain;f=bigtop_toolchain/manifests/libhdfs.pp;hb=refs/heads/BIGTOP-2320]
my own pre-built version of the library, hosted
[here|https://bintray.com/artifact/download/wangzw/deb/dists/trusty/contrib/binary-amd64].
However, this is clearly an insecure and has to be either solved by HAWQ adding
this dependency as the source; or by convincing Bigtop community that hosting
libthrift library is beneficial for the community at large

*Packaging*
- overall, the packaging code is complete and is pushed to the Bigtop branch
(see link below). Considering that the work has been completed about 5 weeks
ago and was aimed at the state of trunk back in the March, there might be some
minor changes, which would require additional tweaks
- libhdfs library code (if already included into HAWQ project) might require
additional changes to the packaging code, so the library can be produces and
properly set in the installation phase
- Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from the
BIGTOP-2320 below)

*Tests*
- smoke tests need to be created (as per BIGTOP-2322), but that seems to be a
minor undertaking once the rest of the work is finished
- packaging tests are required to be integrated into Bigtop stack BIGTOP-2324

*Deployment*
- deployment code is completed. However, it needs to be extended to property
support cluster roles and to be linked to the main {{site.pp}} recipe
- because real-life deployment can not rely on in-house python wrappers using
passwordless-ssh, the lifecycle management and initial bootstrap are done
directly by calling into HAWQ scripts, providing such functionality. It is
possible that some of these interfaces were updated in the last 6 weeks, so
additional testing would be needed.
- it should be responsibility of the HAWQ to provide a concise way of
initializing a master, segment, and so on without a need for password-less ssh,
which is suboptimal and won't be accepted by Bigtop community as it is breaks
the deployment model

*Toolchain*
- toolchain code is completed in the bigtop branch. This will allow to build
HAWQ in the standard Bigtop container available for the CI and 3rd party users
- toolchain code needs to be rebased on top of current Bigtop master. and
possible conflicts would have to be resolved
- once the integration is finished, Bigtop slave images will have to be updated
to enable automatic CI runs


[1] https://issues.apache.org/jira/browse/HAWQ-706

Mime
View raw message