Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <20160425195650.GR7841@tpx>
References: <20160425195650.GR7841@tpx>
From: Radar Da lei <rlei@pivotal.io>
Date: Tue, 26 Apr 2016 15:51:03 +0800
Message-ID: 
 <CAH=zMbpWBbjsW9NfzvmSSfs15DXg_kUvt4vBuXbX+n_ENZLZ6Q@mail.gmail.com>
Subject: Re: HAWQ integration to Apache bigdata stack: remaining steps
To: dev@hawq.incubator.apache.org
Content-Type: multipart/alternative; boundary=f46d04462dae2e02c105315e8f59

--f46d04462dae2e02c105315e8f59
Content-Type: text/plain; charset=UTF-8

Hi Konstantin,

Thanks for list these items out.

For 'External dependencies' part, do you mean 'libthrift' or 'libhdfs'? I
see all the links above point to libhdfs.

    1. If you mean 'libhdfs', now it's already in HAWQ's source code, it is
located in 'depends/libhdfs3', we should build it as the same as libyarn
does.

    2. If you mean thrift, I didn't get what make it different with other
dependencies. Would you please specify the details need to be done?

For "Deployment" part:

    1. Sure we can try to make 'master' and 'segment' to do init/start/stop
without pasword-less. But initialize standby node will require to synchronize
files with master. Any advice how should we handle standby?
        Now HAWQ-469 <https://issues.apache.org/jira/browse/HAWQ-469> is
tracking this, would you share the status, maybe we can assist on this to
speed it up.

    2. Another question is  if "remove password-less" is only required
during hawq installation/initialization(deployment)? Is it required to our
other management tools, e.g. 'hawq config/check/scp/ssh/...', these tools
will not function without password-less.

Thanks.


Regards,
Radar

On Tue, Apr 26, 2016 at 3:56 AM, Konstantin Boudnik <cos@apache.org> wrote:

> guys,
>
> I wanted to put together a list of remaining steps needed before we can
> declare Hawq to be a good citizen of Apache Bigtop (aka Apache bigdata
> stack).
>
> I have put together a JIRA [1] to track these points, and here's the gist
> of
> it for the reader's convenience. Please ping me if you have any questions
> or
> follow up questions.
>
> Regards,
>   Cos
>
> The overview of the remaining steps and the overall status of the
> integration work.
>
> *External dependencies*
> - the biggest issue was and remains the use of libthrift, which isn't
> packaged,
> provided nor supported by anyone. Right now, Bigtop-HAWQ integration branch
> [uses|
> https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob_plain;f=bigtop_toolchain/manifests/libhdfs.pp;hb=refs/heads/BIGTOP-2320
> ]
> my own pre-built version of the library, hosted
> [here|
> https://bintray.com/artifact/download/wangzw/deb/dists/trusty/contrib/binary-amd64
> ].
> However, this is clearly an insecure and has to be either solved by HAWQ
> adding
> this dependency as the source; or by convincing Bigtop community that
> hosting
> libthrift library is beneficial for the community at large
>
> *Packaging*
> - overall, the packaging code is complete and is pushed to the Bigtop
> branch
> (see link below). Considering that the work has been completed about 5
> weeks
> ago and was aimed at the state of trunk back in the March, there might be
> some
> minor changes, which would require additional tweaks
> - libhdfs library code (if already included into HAWQ project) might
> require
> additional changes to the packaging code, so the library can be produces
> and
> properly set in the installation phase
> - Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from the
> BIGTOP-2320 below)
>
> *Tests*
> - smoke tests need to be created (as per BIGTOP-2322), but that seems to
> be a
> minor undertaking once the rest of the work is finished
> - packaging tests are required to be integrated into Bigtop stack
> BIGTOP-2324
>
> *Deployment*
> - deployment code is completed. However, it needs to be extended to
> property
> support cluster roles and to be linked to the main {{site.pp}} recipe
> - because real-life deployment can not rely on in-house python wrappers
> using
> passwordless-ssh, the lifecycle management and initial bootstrap are done
> directly by calling into HAWQ scripts, providing such functionality. It is
> possible that some of these interfaces were updated in the last 6 weeks, so
> additional testing would be needed.
> - it should be responsibility of the HAWQ to provide a concise way of
> initializing a master, segment, and so on without a need for password-less
> ssh,
> which is suboptimal and won't be accepted by Bigtop community as it is
> breaks
> the deployment model
>
> *Toolchain*
> - toolchain code is completed in the bigtop branch. This will allow to
> build
> HAWQ in the standard Bigtop container available for the CI and 3rd party
> users
> - toolchain code needs to be rebased on top of current Bigtop master. and
> possible conflicts would have to be resolved
> - once the integration is finished, Bigtop slave images will have to be
> updated
> to enable automatic CI runs
>
>
> [1] https://issues.apache.org/jira/browse/HAWQ-706
>

--f46d04462dae2e02c105315e8f59--