drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venki Korukanti <venki.koruka...@gmail.com>
Subject Re: Time for a 1.5 release?
Date Thu, 28 Jan 2016 22:59:51 GMT
Here is the partial fix:
https://github.com/vkorukanti/drill/commit/6646b0d08df274a22dff870ad5e6f6914b10fa0b

If the existing StragePlugins are implementing AbstractStoragePlugin class
this fix should resolve the backward compat issue. If they are implementing
StoragePlugin interface directly then it won't resolve the issue. Not sure
how to handle it in that case. Let me know if there any ways to resolve
this issue completely.

Thanks
Venki

On Thu, Jan 28, 2016 at 2:44 PM, Aman Sinha <asinha@maprtech.com> wrote:

> I was able to do the build with maven 3.3.9 (after doing 'brew install
> maven').  Here are my env variables related to maven:
> M2_HOME=/usr/local/Cellar/maven/3.3.9/libexec
> M2=/usr/local/Cellar/maven/3.3.9/libexec/bin
> MAVEN_OPTS=-Xmx2048m -XX:MaxPermSize=256m
>
>
> On Thu, Jan 28, 2016 at 2:20 PM, Jacques Nadeau <jacques@dremio.com>
> wrote:
>
> > Aman, for reference on the Maven version issue. You can see the output of
> > the shade plugin between the two Apache Builds. If you search for
> > maven-shade-plugin you'll see that build 647 (maven 3.0.5) fails because
> it
> > includes a bunch of hadoop (and other libraries that are listed as being
> > excluded). If you look at build 648 (maven 3.3.3), those items are
> included
> > so that the build completes successfully.
> >
> > https://builds.apache.org/job/drill-scm/647/consoleText
> > https://builds.apache.org/job/drill-scm/648/consoleText
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Thu, Jan 28, 2016 at 2:01 PM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Jason,
> > >
> > > As of now I haven't narrowed down the exact commit which caused
> > DRILL-4323
> > > and I do not know the scope of the fix.
> > >
> > > - Rahul
> > >
> > > On Thu, Jan 28, 2016 at 1:39 PM, Jason Altekruse <
> > altekrusejason@gmail.com
> > > >
> > > wrote:
> > >
> > > > I think it makes sense to wait for Venki's fix of the API, as
> > releasing a
> > > > version with the change would make reverting/refactoring the API
> > breaking
> > > > change less useful.
> > > >
> > > > Do you have any idea about the scope of fixing the hive native
> reader,
> > or
> > > > which change since the 1.4 release broke it? Is someone available to
> > look
> > > > at it today?
> > > >
> > > > On Thu, Jan 28, 2016 at 1:33 PM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > > > Just found an issue with hive native parquet reader (DRILL-4323).
> > This
> > > > is a
> > > > > regression from 1.4 and in my opinion should be treated as a
> blocker.
> > > > >
> > > > > - Rahul
> > > > >
> > > > > On Thu, Jan 28, 2016 at 1:12 PM, Venki Korukanti <
> > > > > venki.korukanti@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Sorry I didn't realize I was breaking the public interface API.
> Let
> > > me
> > > > > work
> > > > > > on a patch to make it backward compatible.
> > > > > >
> > > > > > Thanks
> > > > > > Venki
> > > > > >
> > > > > > On Thu, Jan 28, 2016 at 12:54 PM, Steven Phillips <
> > steven@dremio.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I just wanted to bring up an issue that I just now discovered,
> > that
> > > > has
> > > > > > > caused me a fair amount of grief.
> > > > > > >
> > > > > > > https://github.com/apache/drill/pull/300/commits
> > > > > > >
> > > > > > > DRILL-4198 changes a user-facing API, and causes StoragePlugins
> > > that
> > > > > were
> > > > > > > compiled against currently released versions of Drill to
no
> > longer
> > > > > > > functional properly. I would prefer that this breaking
change
> be
> > > > > modified
> > > > > > > to be backward compatible if possible.
> > > > > > >
> > > > > > > On Thu, Jan 28, 2016 at 11:23 AM, Jason Altekruse <
> > > > > > > altekrusejason@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Aman,
> > > > > > > >
> > > > > > > > This is the failure that he was seeing. He figured
out that
> the
> > > new
> > > > > > > > exclusions in jdbc-all were not being respected when
the
> build
> > > was
> > > > > run
> > > > > > > with
> > > > > > > > an older Maven version, causing the jar size to increase
> > > > > significantly.
> > > > > > > He
> > > > > > > > added an enforcer to make sure the JAR didn't grow
> > unexpectedly.
> > > > Can
> > > > > > you
> > > > > > > > try to update your maven version and re-run the build?
> > > > > > > >
> > > > > > > > - Jason
> > > > > > > >
> > > > > > > > On Thu, Jan 28, 2016 at 11:18 AM, Aman Sinha <
> > > asinha@maprtech.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Jacques, I am getting the following build failure
on the
> > latest
> > > > > > master
> > > > > > > > > branch...is this what you saw for the Apache
build ?  My
> mvn
> > > > > version
> > > > > > > > output
> > > > > > > > > is shown below.  Should we all be upgrading to
a newer mvn
> ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [INFO] --- maven-enforcer-plugin:1.3.1:enforce
> > > > > > > > > (enforce-jdbc-jar-compactness) @ drill-jdbc-all
---
> > > > > > > > > [WARNING] Rule 0:
> > > > > org.apache.maven.plugins.enforcer.RequireFilesSize
> > > > > > > > failed
> > > > > > > > > with message:
> > > > > > > > > The file drill-jdbc-all-1.5.0-SNAPSHOT.jar is
outside the
> > > > expected
> > > > > > size
> > > > > > > > > range.
> > > > > > > > >
> > > > > > > > >                   This is likely due to you adding
new
> > > > dependencies
> > > > > > to
> > > > > > > a
> > > > > > > > > java-exec and not updating the excludes in this
module.
> This
> > is
> > > > > > > important
> > > > > > > > > as it minimizes the size of the dependency of
Drill
> > application
> > > > > > users.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > > > > size (44664121) too large. Max. is
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 20000000/Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Administrators-MacBook-Pro-144:incubator-drill
asinha$ mvn
> > > > > --version
> > > > > > > > > Apache Maven 3.0.5
> > (r01de14724cdef164cd33c7c8c2fe155faf9602da;
> > > > > > > 2013-02-19
> > > > > > > > > 05:51:28-0800)
> > > > > > > > > Maven home: /opt/local/share/java/maven3
> > > > > > > > > Java version: 1.7.0_45, vendor: Oracle Corporation
> > > > > > > > > Java home:
> > > > > > > > >
> > > > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
> > > > > > > > > Default locale: en_US, platform encoding: UTF-8
> > > > > > > > > OS name: "mac os x", version: "10.9.5", arch:
"x86_64",
> > family:
> > > > > "mac"
> > > > > > > > >
> > > > > > > > > On Thu, Jan 28, 2016 at 8:20 AM, Jacques Nadeau
<
> > > > > jacques@dremio.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Build back to normal. It looks like the
Apache server was
> > > using
> > > > > an
> > > > > > > old
> > > > > > > > > > version of Maven. Once I switched to something
more
> recent,
> > > the
> > > > > > build
> > > > > > > > > > passed.
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Jacques Nadeau
> > > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > > >
> > > > > > > > > > On Thu, Jan 28, 2016 at 7:02 AM, Jacques
Nadeau <
> > > > > > jacques@dremio.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hmm... this merge caused the Apache
build to fail.
> > > > > > Investigating...
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Jacques Nadeau
> > > > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jan 28, 2016 at 6:31 AM, Jacques
Nadeau <
> > > > > > > jacques@dremio.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> I got clean regression runs as
well. I've merged the
> > > patch.
> > > > > > > > > > >>
> > > > > > > > > > >> Jason, you want to start the release
process?
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> Jacques Nadeau
> > > > > > > > > > >> CTO and Co-Founder, Dremio
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Jan 27, 2016 at 10:42 PM,
Abhishek Girish <
> > > > > > > agirish@mapr.com
> > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Had two clean Functional runs.
TPC-H SF100 was also
> > > > > successful.
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Wed, Jan 27, 2016 at 10:07
PM, rahul challapalli <
> > > > > > > > > > >>> challapallirahul@gmail.com>
wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> > Kicked off a functional
run with your branch. Will
> > let
> > > > you
> > > > > > know
> > > > > > > > > once
> > > > > > > > > > it
> > > > > > > > > > >>> > finishes
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > - Rahul
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Wed, Jan 27, 2016 at
9:56 PM, Jacques Nadeau <
> > > > > > > > > jacques@dremio.com>
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > 4196 was merged today.
I have an updated patch
> for
> > > 4291
> > > > > > that
> > > > > > > is
> > > > > > > > > > >>> ready.
> > > > > > > > > > >>> > > Unfortunately, it
seems that something isn't
> > working
> > > > with
> > > > > > our
> > > > > > > > > > >>> extended
> > > > > > > > > > >>> > > tests so I haven't
been able to run an extended
> > > > > regression.
> > > > > > > > Unit
> > > > > > > > > > >>> tests
> > > > > > > > > > >>> > > pass. Is someone
else possibly able to run a
> > > regression
> > > > > > suite
> > > > > > > > > > against
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > branch [1] so we
can confirm things look good and
> > > start
> > > > > the
> > > > > > > > > release
> > > > > > > > > > >>> > > process?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > thanks,
> > > > > > > > > > >>> > > Jacques
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > [1]
> > > > https://github.com/jacques-n/drill/tree/DRILL-4291v2
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > --
> > > > > > > > > > >>> > > Jacques Nadeau
> > > > > > > > > > >>> > > CTO and Co-Founder,
Dremio
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > On Mon, Jan 25, 2016
at 11:20 AM, Jacques Nadeau
> <
> > > > > > > > > > jacques@dremio.com
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > > I think the
main things are 4196 and 4291
> should
> > be
> > > > > > > > completed.
> > > > > > > > > I
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > > Amit
> > > > > > > > > > >>> > > > was able to
reproduce 4196 locally this weekend
> > so
> > > I
> > > > > > think
> > > > > > > > > we're
> > > > > > > > > > >>> close
> > > > > > > > > > >>> > on
> > > > > > > > > > >>> > > > that.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > --
> > > > > > > > > > >>> > > > Jacques Nadeau
> > > > > > > > > > >>> > > > CTO and Co-Founder,
Dremio
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > On Mon, Jan
25, 2016 at 10:14 AM, Zelaine Fong
> <
> > > > > > > > > > zfong@maprtech.com
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > >> Any updates
on this?  What's blocking us from
> > > taking
> > > > > > this
> > > > > > > > to a
> > > > > > > > > > >>> vote --
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > >> sort merge
join issues?
> > > > > > > > > > >>> > > >>
> > > > > > > > > > >>> > > >> -- Zelaine
> > > > > > > > > > >>> > > >>
> > > > > > > > > > >>> > > >> On Tue,
Jan 19, 2016 at 4:18 PM, Jacques
> Nadeau
> > <
> > > > > > > > > > >>> jacques@dremio.com>
> > > > > > > > > > >>> > > >> wrote:
> > > > > > > > > > >>> > > >>
> > > > > > > > > > >>> > > >> > Bumping
this thread...
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> > Here
are the issues that were mentioned in
> > this
> > > > > thread
> > > > > > > > along
> > > > > > > > > > >>> with a
> > > > > > > > > > >>> > > >> > proposed
categorization:
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> > Release
Blockers
> > > > > > > > > > >>> > > >> > In-progress
Amit
> > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4190
> > > > > > > > > > >>> > > >> > In-progress
Amit
> > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4196
> > > > > > > > > > >>> > > >> > Ready
to merge Jacques
> > > > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4246
> > > > > > > > > > >>> > > >> > In-review
Jinfeng
> > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4256
> > > > > > > > > > >>> > > >> > In-progress
Jacques
> > > > > > > > > > >>> > https://issues.apache.org/jira/browse/DRILL-4278
> > > > > > > > > > >>> > > >> > Ready
to merge Laurent
> > > > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4285
> > > > > > > > > > >>> > > >> > Nice
to Have
> > > > > > > > > > >>> > > >> > Open
Jason/Hakim
> > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4247
> > > > > > > > > > >>> > > >> > In-progress
Jason
> > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4203
> > > > > > > > > > >>> > > >> > Open
Jacques
> > > > > > > > > https://issues.apache.org/jira/browse/DRILL-4266
> > > > > > > > > > >>> > > >> > Ready
to merge Jacques
> > > > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4131
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> > What
do others think? Let's try to get the
> > > > blockers
> > > > > > > > wrapped
> > > > > > > > > up
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > >> next
> > > > > > > > > > >>> > > >> > day
or two and start a release vote...
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> > --
> > > > > > > > > > >>> > > >> > Jacques
Nadeau
> > > > > > > > > > >>> > > >> > CTO
and Co-Founder, Dremio
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> > On
Mon, Jan 4, 2016 at 1:48 PM, Jason
> > Altekruse
> > > <
> > > > > > > > > > >>> > > >> altekrusejason@gmail.com>
> > > > > > > > > > >>> > > >> > wrote:
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >> > >
Hello All,
> > > > > > > > > > >>> > > >> > >
> > > > > > > > > > >>> > > >> > >
With the allocator changes merged and
> about
> > a
> > > > > month
> > > > > > > > since
> > > > > > > > > > the
> > > > > > > > > > >>> last
> > > > > > > > > > >>> > > >> > release
> > > > > > > > > > >>> > > >> > >
I think it would be good to start a vote
> > > soon. I
> > > > > > would
> > > > > > > > > like
> > > > > > > > > > to
> > > > > > > > > > >>> > > >> volunteer
> > > > > > > > > > >>> > > >> > to
> > > > > > > > > > >>> > > >> > >
be release manager.
> > > > > > > > > > >>> > > >> > >
> > > > > > > > > > >>> > > >> > >
I know that there were some issues that
> were
> > > > > > > identified
> > > > > > > > > > after
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > >> > transfer
> > > > > > > > > > >>> > > >> > >
patch was merged. I think that these
> issues
> > > > should
> > > > > > be
> > > > > > > > > fixed
> > > > > > > > > > >>> before
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > >> > cut
a
> > > > > > > > > > >>> > > >> > >
release candidate.
> > > > > > > > > > >>> > > >> > >
> > > > > > > > > > >>> > > >> > >
From looking at the associated JIRAs it
> > looked
> > > > > like
> > > > > > > > there
> > > > > > > > > > was
> > > > > > > > > > >>> a
> > > > > > > > > > >>> > > >> possible
> > > > > > > > > > >>> > > >> > >
short term fix just adjusting the
> > > > > > > > > max_query_memory_per_node
> > > > > > > > > > >>> > option,
> > > > > > > > > > >>> > > >> and
> > > > > > > > > > >>> > > >> > >
some more involved work to change how we
> > > > determine
> > > > > > the
> > > > > > > > > > correct
> > > > > > > > > > >>> > time
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > >> > >
spill during external sort. I believe it
> > makes
> > > > > sense
> > > > > > > to
> > > > > > > > > make
> > > > > > > > > > >>> > > external
> > > > > > > > > > >>> > > >> > sort
> > > > > > > > > > >>> > > >> > >
work well with the newly improved memory
> > > > > accounting
> > > > > > > > before
> > > > > > > > > > >>> > cutting a
> > > > > > > > > > >>> > > >> > >
release, but I'm not sure how much work is
> > > left
> > > > to
> > > > > > be
> > > > > > > > done
> > > > > > > > > > >>> there.
> > > > > > > > > > >>> > > [1]
> > > > > > > > > > >>> > > >> > >
> > > > > > > > > > >>> > > >> > >
Please respond with your thoughts on a
> > release
> > > > > soon
> > > > > > > and
> > > > > > > > > any
> > > > > > > > > > >>> JIRAs
> > > > > > > > > > >>> > > you
> > > > > > > > > > >>> > > >> > would
> > > > > > > > > > >>> > > >> > >
like to include in the release.
> > > > > > > > > > >>> > > >> > >
> > > > > > > > > > >>> > > >> > >
[1] -
> > > > > > > https://issues.apache.org/jira/browse/DRILL-4243
> > > > > > > > > > >>> > > >> > >
> > > > > > > > > > >>> > > >> > >
Thanks,
> > > > > > > > > > >>> > > >> > >
Jason
> > > > > > > > > > >>> > > >> > >
> > > > > > > > > > >>> > > >> >
> > > > > > > > > > >>> > > >>
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> >
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message