hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Denissov <adenis...@pivotal.io>
Subject Re: HAWQ integration to Apache bigdata stack: remaining steps
Date Tue, 26 Apr 2016 18:37:14 GMT
For the deployment and management use cases I think we need to revisit the
design and implementation of the current tools taking into account the
requirements from Ambari and Bigtop management practices.

I would propose 2-prong approach:
- atomic management scripts for operations on individual hosts
- orchestration layer for cluster-level operations

Both Ambari and Bigtop (I'd think) will prefer to deal with scripts on
individual nodes, which will not require SSH keys being distributed for
gpadmin user as the tooling will rely on Ambari or Bigtop key management.
These tools will then orchestrate cross-component operations (like init
standby) via their own wizards or scripts.

For people not using either Ambari or Bigtop, we can provide script-based
orchestration layer similar to cluster-level operations we have currently.
This layer will require ssh keys be exchanged to function properly.

If we agree in principal, I can take a stab into formalizing the Ambari
requirements for the management scripts.

--
Thanks
Alex

On Tue, Apr 26, 2016 at 12:51 AM, Radar Da lei <rlei@pivotal.io> wrote:

> Hi Konstantin,
>
> Thanks for list these items out.
>
> For 'External dependencies' part, do you mean 'libthrift' or 'libhdfs'? I
> see all the links above point to libhdfs.
>
>     1. If you mean 'libhdfs', now it's already in HAWQ's source code, it is
> located in 'depends/libhdfs3', we should build it as the same as libyarn
> does.
>
>     2. If you mean thrift, I didn't get what make it different with other
> dependencies. Would you please specify the details need to be done?
>
> For "Deployment" part:
>
>     1. Sure we can try to make 'master' and 'segment' to do init/start/stop
> without pasword-less. But initialize standby node will require to
> synchronize
> files with master. Any advice how should we handle standby?
>         Now HAWQ-469 <https://issues.apache.org/jira/browse/HAWQ-469> is
> tracking this, would you share the status, maybe we can assist on this to
> speed it up.
>
>     2. Another question is  if "remove password-less" is only required
> during hawq installation/initialization(deployment)? Is it required to our
> other management tools, e.g. 'hawq config/check/scp/ssh/...', these tools
> will not function without password-less.
>
> Thanks.
>
>
>
> Regards,
> Radar
>
> On Tue, Apr 26, 2016 at 3:56 AM, Konstantin Boudnik <cos@apache.org>
> wrote:
>
> > guys,
> >
> > I wanted to put together a list of remaining steps needed before we can
> > declare Hawq to be a good citizen of Apache Bigtop (aka Apache bigdata
> > stack).
> >
> > I have put together a JIRA [1] to track these points, and here's the gist
> > of
> > it for the reader's convenience. Please ping me if you have any questions
> > or
> > follow up questions.
> >
> > Regards,
> >   Cos
> >
> > The overview of the remaining steps and the overall status of the
> > integration work.
> >
> > *External dependencies*
> > - the biggest issue was and remains the use of libthrift, which isn't
> > packaged,
> > provided nor supported by anyone. Right now, Bigtop-HAWQ integration
> branch
> > [uses|
> >
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob_plain;f=bigtop_toolchain/manifests/libhdfs.pp;hb=refs/heads/BIGTOP-2320
> > ]
> > my own pre-built version of the library, hosted
> > [here|
> >
> https://bintray.com/artifact/download/wangzw/deb/dists/trusty/contrib/binary-amd64
> > ].
> > However, this is clearly an insecure and has to be either solved by HAWQ
> > adding
> > this dependency as the source; or by convincing Bigtop community that
> > hosting
> > libthrift library is beneficial for the community at large
> >
> > *Packaging*
> > - overall, the packaging code is complete and is pushed to the Bigtop
> > branch
> > (see link below). Considering that the work has been completed about 5
> > weeks
> > ago and was aimed at the state of trunk back in the March, there might be
> > some
> > minor changes, which would require additional tweaks
> > - libhdfs library code (if already included into HAWQ project) might
> > require
> > additional changes to the packaging code, so the library can be produces
> > and
> > properly set in the installation phase
> > - Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from
> the
> > BIGTOP-2320 below)
> >
> > *Tests*
> > - smoke tests need to be created (as per BIGTOP-2322), but that seems to
> > be a
> > minor undertaking once the rest of the work is finished
> > - packaging tests are required to be integrated into Bigtop stack
> > BIGTOP-2324
> >
> > *Deployment*
> > - deployment code is completed. However, it needs to be extended to
> > property
> > support cluster roles and to be linked to the main {{site.pp}} recipe
> > - because real-life deployment can not rely on in-house python wrappers
> > using
> > passwordless-ssh, the lifecycle management and initial bootstrap are done
> > directly by calling into HAWQ scripts, providing such functionality. It
> is
> > possible that some of these interfaces were updated in the last 6 weeks,
> so
> > additional testing would be needed.
> > - it should be responsibility of the HAWQ to provide a concise way of
> > initializing a master, segment, and so on without a need for
> password-less
> > ssh,
> > which is suboptimal and won't be accepted by Bigtop community as it is
> > breaks
> > the deployment model
> >
> > *Toolchain*
> > - toolchain code is completed in the bigtop branch. This will allow to
> > build
> > HAWQ in the standard Bigtop container available for the CI and 3rd party
> > users
> > - toolchain code needs to be rebased on top of current Bigtop master. and
> > possible conflicts would have to be resolved
> > - once the integration is finished, Bigtop slave images will have to be
> > updated
> > to enable automatic CI runs
> >
> >
> > [1] https://issues.apache.org/jira/browse/HAWQ-706
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message