impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Valencia Serrao" <vser...@us.ibm.com>
Subject Re: Upstreaming ppc64le patches for native-toolchain
Date Thu, 27 Apr 2017 06:13:10 GMT

Hi All,

I have submitted the second patchset of ppc64le-native-toolchain for gerrit
review. Also, I have the build logs of x86 and ppc64le runs of the
native-toolchain ready. Earlier in thread, there was a comment that a new
email list "tests@impala.incubator.apache.org" could be created to share
the same. Is it created already ?

Please let me know the best way I could share the logs for reference.

Regards,
Valencia



From:	Valencia Serrao/Austin/Contr/IBM
To:	Jim Apple <jbapple@cloudera.com>
Cc:	"dev@impala" <dev@impala.incubator.apache.org>, Manish
            Patil/Austin/Contr/IBM@IBMUS, Nishidha
            Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS, David
            Clissold/Austin/IBM@IBMUS, Valencia
            Serrao/Austin/Contr/IBM@IBMUS
Date:	04/20/2017 02:36 PM
Subject:	Re: Upstreaming ppc64le patches for native-toolchain


Hi Jim,

That's OK. Discussing on the testing infra issue, having more clarity on it
and deciding to go ahead with reviewing the ppc64le patches, in this we all
have progressed. I have taken following key points from the earlier
discussion.

1. Jenkins ppc64le testing infra not required for now.
2. ppc64le patch submission can resume, with every patch accompanied with
test results on x86 and ppc64le.
3. All ppc64le-related modifications will be termed as experimental until
they are confirmed to work by community.

Please let me know if I've missed out on anything.

Also, I'm focusing on getting the native-toolchain reviewed and upstreamed
first, then breakpad, kudu, followed by Impala.

Regards,
Valencia




From:	Jim Apple <jbapple@cloudera.com>
To:	"dev@impala" <dev@impala.incubator.apache.org>
Cc:	Manish Patil/Austin/Contr/IBM@IBMUS, Nishidha
            Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS, Valencia
            Serrao/Austin/Contr/IBM@IBMUS
Date:	04/20/2017 03:53 AM
Subject:	Re: Upstreaming ppc64le patches for native-toolchain



OK, then it sounds like for now the ppc64le work can proceed for now
without a Jenkins testing available to the whole community, unless anyone
else objects.

Valencia et al.: apologies if I delayed your progress

Silvius & Tim: thanks for the good suggestions.

On Mon, Apr 17, 2017 at 4:19 PM, Tim Armstrong <tarmstrong@cloudera.com>
wrote:
  Makes sense to me

  On Mon, Apr 17, 2017 at 3:11 PM, Jim Apple <jbapple@cloudera.com> wrote:

  > I'm OK with us calling this "experimental" (once the patches are in and
  we
  > think it works) and letting patches continue to land with our current
  > testing regime.
  >
  > Later, if we add daily test reports with contributor-managed hardware
  or
  > even pre-commit testing on jenkins.impala.io in a VM, maybe we could
  > upgrade to other names than "experimental", like "beta" or
  "best-effort",
  > if we still want to have caveats.
  >
  > How does that sound to everyone else?
  >
  > On Mon, Apr 17, 2017 at 2:54 PM, Tim Armstrong <tarmstrong@cloudera.com
  >
  > wrote:
  >
  > > Something like that makes sense - I think it should be an
  experimental or
  > > unsupported feature until if/when we have a critical mass of
  committers
  > to
  > > maintain it.
  > >
  > > On Mon, Apr 17, 2017 at 2:46 PM, Jim Apple <jbapple@cloudera.com>
  wrote:
  > >
  > > > That makes sense to me. It would be good for us to provide support
  > > without
  > > > completely focusing all development effort on a HW platform with
  fewer
  > > > users.
  > > >
  > > > If ppc64le support lands in 2.10, and between 2.10 and 2.11 the
  ppc64le
  > > > tests start breaking for reasons nobody with HW access can debug,
  > should
  > > we
  > > > say in 2.11 release notes that ppc64le is no longer supported?
  > > >
  > > > Or perhaps, even in the first release where we think it works, we
  > should
  > > > spell it out as a platform with only "best-effort" support, that
  way we
  > > > don't have to retract support?
  > > >
  > > > On Mon, Apr 17, 2017 at 2:41 PM, Marcel Kornacker <
  marcel@cloudera.com
  > >
  > > > wrote:
  > > >
  > > > > My main concern is that we don't unduly burden the development
  > process.
  > > > As
  > > > > such, it makes a lot of sense to keep the PPC tests out of the
  > regular
  > > > > tests and have the party that's interested in the PPC tests
  create
  > the
  > > > > infrastructure and run those tests.
  > > > >
  > > > > On Mon, Apr 17, 2017 at 2:30 PM, Jim Apple <jbapple@cloudera.com>
  > > wrote:
  > > > >
  > > > > > One concern I have is sustainability. If only one Impala
  > contributor
  > > > can
  > > > > > work with ppc64le, and that contributor is not as seasoned as
  some
  > of
  > > > the
  > > > > > other committers, what happens if ppc64le breaks and the one
  person
  > > > with
  > > > > VM
  > > > > > access can't fix it?
  > > > > >
  > > > > > Part of my concern is just how flaky the current tests are,
  too. It
  > > > takes
  > > > > > some time to be able to distinguish broken tests that are flaky
  and
  > > > > broken
  > > > > > tests that are the result of a specific commit.
  > > > > >
  > > > > > My hope was that with a ppc64le VM (maybe through Qemu?) that
  runs
  > on
  > > > > > x86-64 Linux, the other contributors could help fix things that
  > broke
  > > > on
  > > > > > that platform.
  > > > > >
  > > > > > On Mon, Apr 17, 2017 at 12:51 PM, Marcel Kornacker <
  > > > marcel@cloudera.com>
  > > > > > wrote:
  > > > > >
  > > > > > > +1
  > > > > > >
  > > > > > > On Mon, Apr 17, 2017 at 11:56 AM, Henry Robinson <
  > > henry@cloudera.com
  > > > >
  > > > > > > wrote:
  > > > > > >
  > > > > > > > +1
  > > > > > > >
  > > > > > > > On Mon, Apr 17, 2017 at 9:09 AM Tim Armstrong <
  > > > > tarmstrong@cloudera.com
  > > > > > >
  > > > > > > > wrote:
  > > > > > > >
  > > > > > > > > I feel like we shouldn't make PPC part of pre-commit
at
  least
  > > > > > > initially -
  > > > > > > > > it's an unreasonable barrier if contributors/committers
  to
  > > debug
  > > > > > issues
  > > > > > > > on
  > > > > > > > > a platform they don't have easy access to. Having
the
  testing
  > > > infra
  > > > > > is
  > > > > > > > > still important because we don't want to have code
in
  there
  > > that
  > > > > will
  > > > > > > > > gradually bit-rot without us noticing.
  > > > > > > > >
  > > > > > > > > On Mon, Apr 17, 2017 at 8:51 AM, Silvius Rus <
  > > srus@cloudera.com>
  > > > > > > wrote:
  > > > > > > > >
  > > > > > > > > > Would it make sense to _not_ run PPC tests
as part of
  > > > presubmit?
  > > > > > > > Instead
  > > > > > > > > > Valencia could set up nightly tests using in-house
  > > > > infrastructure.
  > > > > > > And
  > > > > > > > > > share the test results, e.g., by sending them
to a new
  > email
  > > > list
  > > > > > > > > > tests@impala.incubator.apache.org (that we'd
need to
  > create)
  > > > so
  > > > > > > > everyone
  > > > > > > > > > can see when there are failures or if coverage
stops
  for
  > > > whatever
  > > > > > > > reason.
  > > > > > > > > > GCC has been doing something like this for
long,
  > > > > > > > > > https://gcc.gnu.org/ml/gcc-testresults/2017-04/.
  > > > > > > > > >
  > > > > > > > > > On Tue, Apr 11, 2017 at 9:44 AM, Jim Apple
<
  > > > jbapple@cloudera.com
  > > > > >
  > > > > > > > wrote:
  > > > > > > > > >
  > > > > > > > > > > >
  > > > > > > > > > > > Locally, I work on native-toolchain
using a VM
  > configured
  > > > > with
  > > > > > > > > > > > Ubuntu16.04ppc64le, 4GB RAM and 50GB
of HDD. If  we
  > > provide
  > > > > > you a
  > > > > > > > VM
  > > > > > > > > > with
  > > > > > > > > > > > this config, will it be sufficient
?
  > > > > > > > > > > >
  > > > > > > > > > >
  > > > > > > > > > > What hypervisor/emulator will it use?
  > > > > > > > > > >
  > > > > > > > > > > What are the requirements of the host
OS and host
  > hardware?
  > > > > > > > > > >
  > > > > > > > > > > Why is the config you have it set to so
important
  that
  > you
  > > > > > mention
  > > > > > > it
  > > > > > > > > in
  > > > > > > > > > > your email - will the config be locked
down into that
  > > config
  > > > or
  > > > > > can
  > > > > > > > it
  > > > > > > > > be
  > > > > > > > > > > reconfigured later?
  > > > > > > > > > >
  > > > > > > > > > > How is the VM controlled from the host
OS? Keep in
  mind
  > > that
  > > > a
  > > > > > GUI
  > > > > > > > > cannot
  > > > > > > > > > > be the only option for automated tests.
  > > > > > > > > > >
  > > > > > > > > > > FWIW, Impala's test suite probably cannot
fully
  complete
  > > > > without
  > > > > > at
  > > > > > > > > least
  > > > > > > > > > > 8, and I suspect 16, GB of RAM, and we
might need
  more
  > disk
  > > > > > space,
  > > > > > > > too,
  > > > > > > > > > but
  > > > > > > > > > > these should be reconfigurable with most
  > > > hypervisors/emulators.
  > > > > > > > > > >
  > > > > > > > > >
  > > > > > > > >
  > > > > > > > --
  > > > > > > > Henry Robinson
  > > > > > > > Software Engineer
  > > > > > > > Cloudera
  > > > > > > > 415-994-6679
  > > > > > > >
  > > > > > >
  > > > > >
  > > > >
  > > >
  > >
  >




Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message