impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Armstrong <tarmstr...@cloudera.com>
Subject Re: Toolchain - versioning dependencies with the same version number
Date Tue, 28 Feb 2017 04:51:00 GMT
Maybe I'm misunderstanding, but wouldn't that force re-downloading of the
entire toolchain every time a developer switches between branches with
different build IDs?

I know some developers do that frequently, e.g. to try and reproduce bugs
on older versions or backport patches.

I agree it would be good to fix this, since I've run into this problem
before, I'm just not quite sure what the best solution is. In the other
case where I had this issue with LLVM I changed the version number (by
appending noasserts-) to it, but that's really just a hack.

-Tim

On Mon, Feb 27, 2017 at 4:35 PM, Henry Robinson <henry@cloudera.com> wrote:

> As Matt said, I have a patch that implements build ID-based versioning at
> https://gerrit.cloudera.org/#/c/6166/2.
>
> Does anyone want to take a look? If we could get this in soon it would help
> smooth over the LZ4 change which is going in shortly.
>
> On 27 February 2017 at 14:21, Henry Robinson <henry@cloudera.com> wrote:
>
> > I agree that that might be useful, and that it's a separately addressable
> > problem.
> >
> > On 27 February 2017 at 14:18, Matthew Jacobs <mj@cloudera.com> wrote:
> >
> >> Just catching up to this e-mail, though I had seen your code reviews
> >> and I think this approach makes sense. An additional concern would be
> >> how to identify how a toolchain package was built, and AFAIK this is
> >> tricky now if only the 'toolchain ID' is known. Before I saw this
> >> e-mail I was thinking about this problem (which I think we can address
> >> separately), and that we might want to write the native-toolchain git
> >> hash with every toolchain build so that the exact build scripts are
> >> associated with those build artifacts. I filed
> >> https://issues.cloudera.org/browse/IMPALA-5002 for this related
> >> problem.
> >>
> >> On Sat, Feb 25, 2017 at 10:22 PM, Henry Robinson <henry@apache.org>
> >> wrote:
> >> > As written, the toolchain can't apparently deal with the possibility
> of
> >> > build flags changing, but a dependency version remaining the same.
> >> >
> >> > LZ4 has never (afaict) been built with optimization enabled. I have a
> >> > commit that enables -O3, but that continues to produce artifacts for
> >> > lz4-1.7.5 with no version change. This is a problem because
> >> bootstrapping
> >> > the toolchain will fail to pick up the new binaries - because the
> >> > previously downloaded version is still in the local cache, and won't
> be
> >> > overwritten because of the version change.
> >> >
> >> > I think the simplest way to fix this is to write the toolchain build
> ID
> >> to
> >> > the dependency version file (that's in the local cache only) when it's
> >> > downloaded. If that ID changes, the dependency will be re-downloaded.
> >> >
> >> > This has the disadvantage that any bump in IMPALA_TOOLCHAIN_BUILD_ID
> >> will
> >> > invalidate all dependencies, and bin/bootstrap_toolchain.py will
> >> > re-download all of them. My feeling is that that cost is better than
> >> trying
> >> > to individually determine whether a dependency has changed between
> >> > toolchain builds.
> >> >
> >> > Any thoughts on whether this is the right way to go?
> >> >
> >> > Henry
> >>
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679 <(415)%20994-6679>
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message