hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: HAWQ integration to Apache bigdata stack: remaining steps
Date Fri, 29 Apr 2016 18:12:15 GMT
On Tue, Apr 26, 2016 at 03:51PM, Radar Da lei wrote:
> Hi Konstantin,
> 
> Thanks for list these items out.
> 
> For 'External dependencies' part, do you mean 'libthrift' or 'libhdfs'? I
> see all the links above point to libhdfs.
> 
>     1. If you mean 'libhdfs', now it's already in HAWQ's source code, it is
> located in 'depends/libhdfs3', we should build it as the same as libyarn
> does.
> 
>     2. If you mean thrift, I didn't get what make it different with other
> dependencies. Would you please specify the details need to be done?

Sorry, I was taking about libthrift. The points about libhdfs are no longer
valid indeed, as it has been moved into the project codebase.

> For "Deployment" part:
> 
>     1. Sure we can try to make 'master' and 'segment' to do init/start/stop
> without pasword-less. But initialize standby node will require to synchronize
> files with master. Any advice how should we handle standby?
>         Now HAWQ-469 <https://issues.apache.org/jira/browse/HAWQ-469> is
> tracking this, would you share the status, maybe we can assist on this to
> speed it up.

This is fine. We doing something similar when standing up HDFS HA, so there's
no technical blocker in this.

>     2. Another question is  if "remove password-less" is only required
> during hawq installation/initialization(deployment)? Is it required to our
> other management tools, e.g. 'hawq config/check/scp/ssh/...', these tools
> will not function without password-less.

I am not asking to remove per se, but rather to have a basic set of scripts
that would work at the node level only, and then wrap them into shh-dependant
logic where you see fit.

Cos

> On Tue, Apr 26, 2016 at 3:56 AM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > guys,
> >
> > I wanted to put together a list of remaining steps needed before we can
> > declare Hawq to be a good citizen of Apache Bigtop (aka Apache bigdata
> > stack).
> >
> > I have put together a JIRA [1] to track these points, and here's the gist
> > of
> > it for the reader's convenience. Please ping me if you have any questions
> > or
> > follow up questions.
> >
> > Regards,
> >   Cos
> >
> > The overview of the remaining steps and the overall status of the
> > integration work.
> >
> > *External dependencies*
> > - the biggest issue was and remains the use of libthrift, which isn't
> > packaged,
> > provided nor supported by anyone. Right now, Bigtop-HAWQ integration branch
> > [uses|
> > https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob_plain;f=bigtop_toolchain/manifests/libhdfs.pp;hb=refs/heads/BIGTOP-2320
> > ]
> > my own pre-built version of the library, hosted
> > [here|
> > https://bintray.com/artifact/download/wangzw/deb/dists/trusty/contrib/binary-amd64
> > ].
> > However, this is clearly an insecure and has to be either solved by HAWQ
> > adding
> > this dependency as the source; or by convincing Bigtop community that
> > hosting
> > libthrift library is beneficial for the community at large
> >
> > *Packaging*
> > - overall, the packaging code is complete and is pushed to the Bigtop
> > branch
> > (see link below). Considering that the work has been completed about 5
> > weeks
> > ago and was aimed at the state of trunk back in the March, there might be
> > some
> > minor changes, which would require additional tweaks
> > - libhdfs library code (if already included into HAWQ project) might
> > require
> > additional changes to the packaging code, so the library can be produces
> > and
> > properly set in the installation phase
> > - Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from the
> > BIGTOP-2320 below)
> >
> > *Tests*
> > - smoke tests need to be created (as per BIGTOP-2322), but that seems to
> > be a
> > minor undertaking once the rest of the work is finished
> > - packaging tests are required to be integrated into Bigtop stack
> > BIGTOP-2324
> >
> > *Deployment*
> > - deployment code is completed. However, it needs to be extended to
> > property
> > support cluster roles and to be linked to the main {{site.pp}} recipe
> > - because real-life deployment can not rely on in-house python wrappers
> > using
> > passwordless-ssh, the lifecycle management and initial bootstrap are done
> > directly by calling into HAWQ scripts, providing such functionality. It is
> > possible that some of these interfaces were updated in the last 6 weeks, so
> > additional testing would be needed.
> > - it should be responsibility of the HAWQ to provide a concise way of
> > initializing a master, segment, and so on without a need for password-less
> > ssh,
> > which is suboptimal and won't be accepted by Bigtop community as it is
> > breaks
> > the deployment model
> >
> > *Toolchain*
> > - toolchain code is completed in the bigtop branch. This will allow to
> > build
> > HAWQ in the standard Bigtop container available for the CI and 3rd party
> > users
> > - toolchain code needs to be rebased on top of current Bigtop master. and
> > possible conflicts would have to be resolved
> > - once the integration is finished, Bigtop slave images will have to be
> > updated
> > to enable automatic CI runs
> >
> >
> > [1] https://issues.apache.org/jira/browse/HAWQ-706
> >

Mime
View raw message