impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Apple <jbap...@cloudera.com>
Subject Re: Upstreaming ppc64le patches for native-toolchain
Date Thu, 27 Apr 2017 15:07:30 GMT
I do not think this is appropriate yet because of how early in the process
we are.

On Wed, Apr 26, 2017 at 11:13 PM, Valencia Serrao <vserrao@us.ibm.com>
wrote:

> Hi All,
>
> I have submitted the second patchset of ppc64le-native-toolchain for
> gerrit review. Also, I have the build logs of x86 and ppc64le runs of the
> native-toolchain ready. Earlier in thread, there was a comment that a new
> email list "*tests@impala.incubator.apache.org*
> <tests@impala.incubator.apache.org>" could be created to share the same.
> Is it created already ?
>
> Please let me know the best way I could share the logs for reference.
>
> Regards,
> Valencia
>
> [image: Inactive hide details for Valencia Serrao---04/20/2017 02:36:33
> PM---Hi Jim, That's OK. Discussing on the testing infra issue,]Valencia
> Serrao---04/20/2017 02:36:33 PM---Hi Jim, That's OK. Discussing on the
> testing infra issue, having more clarity on it and deciding to
>
> From: Valencia Serrao/Austin/Contr/IBM
> To: Jim Apple <jbapple@cloudera.com>
> Cc: "dev@impala" <dev@impala.incubator.apache.org>, Manish
> Patil/Austin/Contr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS,
> Sudarshan Jagadale/Austin/Contr/IBM@IBMUS, David Clissold/Austin/IBM@IBMUS,
> Valencia Serrao/Austin/Contr/IBM@IBMUS
> Date: 04/20/2017 02:36 PM
> Subject: Re: Upstreaming ppc64le patches for native-toolchain
> ------------------------------
>
>
> Hi Jim,
>
> That's OK. Discussing on the testing infra issue, having more clarity on
> it and deciding to go ahead with reviewing the ppc64le patches, in this we
> all have progressed. I have taken following key points from the earlier
> discussion.
>
> 1. Jenkins ppc64le testing infra not required for now.
> 2. ppc64le patch submission can resume, with every patch accompanied with
> test results on x86 and ppc64le.
> 3. All ppc64le-related modifications will be termed as experimental until
> they are confirmed to work by community.
>
> Please let me know if I've missed out on anything.
>
> Also, I'm focusing on getting the native-toolchain reviewed and upstreamed
> first, then breakpad, kudu, followed by Impala.
>
> Regards,
> Valencia
>
>
> [image: Inactive hide details for Jim Apple ---04/20/2017 03:53:19
> AM---OK, then it sounds like for now the ppc64le work can proceed fo]Jim
> Apple ---04/20/2017 03:53:19 AM---OK, then it sounds like for now the
> ppc64le work can proceed for now without a Jenkins testing avail
>
> From: Jim Apple <jbapple@cloudera.com>
> To: "dev@impala" <dev@impala.incubator.apache.org>
> Cc: Manish Patil/Austin/Contr/IBM@IBMUS, Nishidha
> Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS,
> Valencia Serrao/Austin/Contr/IBM@IBMUS
> Date: 04/20/2017 03:53 AM
> Subject: Re: Upstreaming ppc64le patches for native-toolchain
> ------------------------------
>
>
>
> OK, then it sounds like for now the ppc64le work can proceed for now
> without a Jenkins testing available to the whole community, unless anyone
> else objects.
>
> Valencia et al.: apologies if I delayed your progress
>
> Silvius & Tim: thanks for the good suggestions.
>
> On Mon, Apr 17, 2017 at 4:19 PM, Tim Armstrong <*tarmstrong@cloudera.com*
> <tarmstrong@cloudera.com>> wrote:
>
>    Makes sense to me
>
>    On Mon, Apr 17, 2017 at 3:11 PM, Jim Apple <*jbapple@cloudera.com*
>    <jbapple@cloudera.com>> wrote:
>
>    > I'm OK with us calling this "experimental" (once the patches are in
>    and we
>    > think it works) and letting patches continue to land with our current
>    > testing regime.
>    >
>    > Later, if we add daily test reports with contributor-managed
>    hardware or
>    > even pre-commit testing on *jenkins.impala.io*
>    <http://jenkins.impala.io/> in a VM, maybe we could
>    > upgrade to other names than "experimental", like "beta" or
>    "best-effort",
>    > if we still want to have caveats.
>    >
>    > How does that sound to everyone else?
>    >
>    > On Mon, Apr 17, 2017 at 2:54 PM, Tim Armstrong <
>    *tarmstrong@cloudera.com* <tarmstrong@cloudera.com>>
>    > wrote:
>    >
>    > > Something like that makes sense - I think it should be an
>    experimental or
>    > > unsupported feature until if/when we have a critical mass of
>    committers
>    > to
>    > > maintain it.
>    > >
>    > > On Mon, Apr 17, 2017 at 2:46 PM, Jim Apple <*jbapple@cloudera.com*
>    <jbapple@cloudera.com>> wrote:
>    > >
>    > > > That makes sense to me. It would be good for us to provide
>    support
>    > > without
>    > > > completely focusing all development effort on a HW platform with
>    fewer
>    > > > users.
>    > > >
>    > > > If ppc64le support lands in 2.10, and between 2.10 and 2.11 the
>    ppc64le
>    > > > tests start breaking for reasons nobody with HW access can debug,
>    > should
>    > > we
>    > > > say in 2.11 release notes that ppc64le is no longer supported?
>    > > >
>    > > > Or perhaps, even in the first release where we think it works, we
>    > should
>    > > > spell it out as a platform with only "best-effort" support, that
>    way we
>    > > > don't have to retract support?
>    > > >
>    > > > On Mon, Apr 17, 2017 at 2:41 PM, Marcel Kornacker <
>    *marcel@cloudera.com* <marcel@cloudera.com>
>    > >
>    > > > wrote:
>    > > >
>    > > > > My main concern is that we don't unduly burden the development
>    > process.
>    > > > As
>    > > > > such, it makes a lot of sense to keep the PPC tests out of the
>    > regular
>    > > > > tests and have the party that's interested in the PPC tests
>    create
>    > the
>    > > > > infrastructure and run those tests.
>    > > > >
>    > > > > On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple <
>    *jbapple@cloudera.com* <jbapple@cloudera.com>>
>    > > wrote:
>    > > > >
>    > > > > > One concern I have is sustainability. If only one Impala
>    > contributor
>    > > > can
>    > > > > > work with ppc64le, and that contributor is not as seasoned
>    as some
>    > of
>    > > > the
>    > > > > > other committers, what happens if ppc64le breaks and the
one
>    person
>    > > > with
>    > > > > VM
>    > > > > > access can't fix it?
>    > > > > >
>    > > > > > Part of my concern is just how flaky the current tests are,
>    too. It
>    > > > takes
>    > > > > > some time to be able to distinguish broken tests that are
>    flaky and
>    > > > > broken
>    > > > > > tests that are the result of a specific commit.
>    > > > > >
>    > > > > > My hope was that with a ppc64le VM (maybe through Qemu?)
>    that runs
>    > on
>    > > > > > x86-64 Linux, the other contributors could help fix things
>    that
>    > broke
>    > > > on
>    > > > > > that platform.
>    > > > > >
>    > > > > > On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker <
>    > > > *marcel@cloudera.com* <marcel@cloudera.com>>
>    > > > > > wrote:
>    > > > > >
>    > > > > > > +1
>    > > > > > >
>    > > > > > > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson <
>    > > *henry@cloudera.com* <henry@cloudera.com>
>    > > > >
>    > > > > > > wrote:
>    > > > > > >
>    > > > > > > > +1
>    > > > > > > >
>    > > > > > > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong <
>    > > > > *tarmstrong@cloudera.com* <tarmstrong@cloudera.com>
>    > > > > > >
>    > > > > > > > wrote:
>    > > > > > > >
>    > > > > > > > > I feel like we shouldn't make PPC part of
pre-commit
>    at least
>    > > > > > > initially -
>    > > > > > > > > it's an unreasonable barrier if
>    contributors/committers to
>    > > debug
>    > > > > > issues
>    > > > > > > > on
>    > > > > > > > > a platform they don't have easy access to.
Having the
>    testing
>    > > > infra
>    > > > > > is
>    > > > > > > > > still important because we don't want to have
code in
>    there
>    > > that
>    > > > > will
>    > > > > > > > > gradually bit-rot without us noticing.
>    > > > > > > > >
>    > > > > > > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus
<
>    > > *srus@cloudera.com* <srus@cloudera.com>>
>    > > > > > > wrote:
>    > > > > > > > >
>    > > > > > > > > > Would it make sense to _not_ run PPC
tests as part of
>    > > > presubmit?
>    > > > > > > > Instead
>    > > > > > > > > > Valencia could set up nightly tests using
in-house
>    > > > > infrastructure.
>    > > > > > > And
>    > > > > > > > > > share the test results, e.g., by sending
them to a
>    new
>    > email
>    > > > list
>    > > > > > > > > > *tests@impala.incubator.apache.org*
>    <tests@impala.incubator.apache.org> (that we'd need to
>    > create)
>    > > > so
>    > > > > > > > everyone
>    > > > > > > > > > can see when there are failures or if
coverage stops
>    for
>    > > > whatever
>    > > > > > > > reason.
>    > > > > > > > > > GCC has been doing something like this
for long,
>    > > > > > > > > > *https://gcc.gnu.org/ml/gcc-testresults/2017-04/*
>    <https://gcc.gnu.org/ml/gcc-testresults/2017-04/>.
>    > > > > > > > > >
>    > > > > > > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim
Apple <
>    > > > *jbapple@cloudera.com* <jbapple@cloudera.com>
>    > > > > >
>    > > > > > > > wrote:
>    > > > > > > > > >
>    > > > > > > > > > > >
>    > > > > > > > > > > > Locally, I work on native-toolchain
using a VM
>    > configured
>    > > > > with
>    > > > > > > > > > > > Ubuntu16.04ppc64le, 4GB RAM
and 50GB of HDD. If
>    we
>    > > provide
>    > > > > > you a
>    > > > > > > > VM
>    > > > > > > > > > with
>    > > > > > > > > > > > this config, will it be sufficient
?
>    > > > > > > > > > > >
>    > > > > > > > > > >
>    > > > > > > > > > > What hypervisor/emulator will it
use?
>    > > > > > > > > > >
>    > > > > > > > > > > What are the requirements of the
host OS and host
>    > hardware?
>    > > > > > > > > > >
>    > > > > > > > > > > Why is the config you have it set
to so important
>    that
>    > you
>    > > > > > mention
>    > > > > > > it
>    > > > > > > > > in
>    > > > > > > > > > > your email - will the config be
locked down into
>    that
>    > > config
>    > > > or
>    > > > > > can
>    > > > > > > > it
>    > > > > > > > > be
>    > > > > > > > > > > reconfigured later?
>    > > > > > > > > > >
>    > > > > > > > > > > How is the VM controlled from the
host OS? Keep in
>    mind
>    > > that
>    > > > a
>    > > > > > GUI
>    > > > > > > > > cannot
>    > > > > > > > > > > be the only option for automated
tests.
>    > > > > > > > > > >
>    > > > > > > > > > > FWIW, Impala's test suite probably
cannot fully
>    complete
>    > > > > without
>    > > > > > at
>    > > > > > > > > least
>    > > > > > > > > > > 8, and I suspect 16, GB of RAM,
and we might need
>    more
>    > disk
>    > > > > > space,
>    > > > > > > > too,
>    > > > > > > > > > but
>    > > > > > > > > > > these should be reconfigurable with
most
>    > > > hypervisors/emulators.
>    > > > > > > > > > >
>    > > > > > > > > >
>    > > > > > > > >
>    > > > > > > > --
>    > > > > > > > Henry Robinson
>    > > > > > > > Software Engineer
>    > > > > > > > Cloudera
>    > > > > > > > *415-994-6679* <415-994-6679>
>    > > > > > > >
>    > > > > > >
>    > > > > >
>    > > > >
>    > > >
>    > >
>    >
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message