hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lei Chang <lei_ch...@apache.org>
Subject Re: HAWQ integration to Apache bigdata stack: remaining steps
Date Wed, 27 Apr 2016 00:29:19 GMT
+1 for the proposal. Formalizing the requirements from Bigtop and Ambari
sounds a good start point.

And It looks making bigtop work in short term without exchange ssh keys
only needs a few small changes in the standby script if bigtop takes the
responsibility to copy the files between master and standby as a
precondition.

Cheers
Lei


On Wed, Apr 27, 2016 at 2:37 AM, Alexander Denissov <adenissov@pivotal.io>
wrote:

> For the deployment and management use cases I think we need to revisit the
> design and implementation of the current tools taking into account the
> requirements from Ambari and Bigtop management practices.
>
> I would propose 2-prong approach:
> - atomic management scripts for operations on individual hosts
> - orchestration layer for cluster-level operations
>
> Both Ambari and Bigtop (I'd think) will prefer to deal with scripts on
> individual nodes, which will not require SSH keys being distributed for
> gpadmin user as the tooling will rely on Ambari or Bigtop key management.
> These tools will then orchestrate cross-component operations (like init
> standby) via their own wizards or scripts.
>
> For people not using either Ambari or Bigtop, we can provide script-based
> orchestration layer similar to cluster-level operations we have currently.
> This layer will require ssh keys be exchanged to function properly.
>
> If we agree in principal, I can take a stab into formalizing the Ambari
> requirements for the management scripts.
>
> --
> Thanks
> Alex
>
> On Tue, Apr 26, 2016 at 12:51 AM, Radar Da lei <rlei@pivotal.io> wrote:
>
> > Hi Konstantin,
> >
> > Thanks for list these items out.
> >
> > For 'External dependencies' part, do you mean 'libthrift' or 'libhdfs'? I
> > see all the links above point to libhdfs.
> >
> >     1. If you mean 'libhdfs', now it's already in HAWQ's source code, it
> is
> > located in 'depends/libhdfs3', we should build it as the same as libyarn
> > does.
> >
> >     2. If you mean thrift, I didn't get what make it different with other
> > dependencies. Would you please specify the details need to be done?
> >
> > For "Deployment" part:
> >
> >     1. Sure we can try to make 'master' and 'segment' to do
> init/start/stop
> > without pasword-less. But initialize standby node will require to
> > synchronize
> > files with master. Any advice how should we handle standby?
> >         Now HAWQ-469 <https://issues.apache.org/jira/browse/HAWQ-469> is
> > tracking this, would you share the status, maybe we can assist on this to
> > speed it up.
> >
> >     2. Another question is  if "remove password-less" is only required
> > during hawq installation/initialization(deployment)? Is it required to
> our
> > other management tools, e.g. 'hawq config/check/scp/ssh/...', these tools
> > will not function without password-less.
> >
> > Thanks.
> >
> >
> >
> > Regards,
> > Radar
> >
> > On Tue, Apr 26, 2016 at 3:56 AM, Konstantin Boudnik <cos@apache.org>
> > wrote:
> >
> > > guys,
> > >
> > > I wanted to put together a list of remaining steps needed before we can
> > > declare Hawq to be a good citizen of Apache Bigtop (aka Apache bigdata
> > > stack).
> > >
> > > I have put together a JIRA [1] to track these points, and here's the
> gist
> > > of
> > > it for the reader's convenience. Please ping me if you have any
> questions
> > > or
> > > follow up questions.
> > >
> > > Regards,
> > >   Cos
> > >
> > > The overview of the remaining steps and the overall status of the
> > > integration work.
> > >
> > > *External dependencies*
> > > - the biggest issue was and remains the use of libthrift, which isn't
> > > packaged,
> > > provided nor supported by anyone. Right now, Bigtop-HAWQ integration
> > branch
> > > [uses|
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob_plain;f=bigtop_toolchain/manifests/libhdfs.pp;hb=refs/heads/BIGTOP-2320
> > > ]
> > > my own pre-built version of the library, hosted
> > > [here|
> > >
> >
> https://bintray.com/artifact/download/wangzw/deb/dists/trusty/contrib/binary-amd64
> > > ].
> > > However, this is clearly an insecure and has to be either solved by
> HAWQ
> > > adding
> > > this dependency as the source; or by convincing Bigtop community that
> > > hosting
> > > libthrift library is beneficial for the community at large
> > >
> > > *Packaging*
> > > - overall, the packaging code is complete and is pushed to the Bigtop
> > > branch
> > > (see link below). Considering that the work has been completed about 5
> > > weeks
> > > ago and was aimed at the state of trunk back in the March, there might
> be
> > > some
> > > minor changes, which would require additional tweaks
> > > - libhdfs library code (if already included into HAWQ project) might
> > > require
> > > additional changes to the packaging code, so the library can be
> produces
> > > and
> > > properly set in the installation phase
> > > - Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from
> > the
> > > BIGTOP-2320 below)
> > >
> > > *Tests*
> > > - smoke tests need to be created (as per BIGTOP-2322), but that seems
> to
> > > be a
> > > minor undertaking once the rest of the work is finished
> > > - packaging tests are required to be integrated into Bigtop stack
> > > BIGTOP-2324
> > >
> > > *Deployment*
> > > - deployment code is completed. However, it needs to be extended to
> > > property
> > > support cluster roles and to be linked to the main {{site.pp}} recipe
> > > - because real-life deployment can not rely on in-house python wrappers
> > > using
> > > passwordless-ssh, the lifecycle management and initial bootstrap are
> done
> > > directly by calling into HAWQ scripts, providing such functionality. It
> > is
> > > possible that some of these interfaces were updated in the last 6
> weeks,
> > so
> > > additional testing would be needed.
> > > - it should be responsibility of the HAWQ to provide a concise way of
> > > initializing a master, segment, and so on without a need for
> > password-less
> > > ssh,
> > > which is suboptimal and won't be accepted by Bigtop community as it is
> > > breaks
> > > the deployment model
> > >
> > > *Toolchain*
> > > - toolchain code is completed in the bigtop branch. This will allow to
> > > build
> > > HAWQ in the standard Bigtop container available for the CI and 3rd
> party
> > > users
> > > - toolchain code needs to be rebased on top of current Bigtop master.
> and
> > > possible conflicts would have to be resolved
> > > - once the integration is finished, Bigtop slave images will have to be
> > > updated
> > > to enable automatic CI runs
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/HAWQ-706
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message