drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chun Chang <cch...@maprtech.com>
Subject Re: Time for the 1.6 Release
Date Sat, 05 Mar 2016 01:45:51 GMT
Jacques submitted a PR for fixing the failed baselines. I've merged them
into automation master and confirmed the failed tests are all passing now.
Thanks.

-Chun


On Thu, Mar 3, 2016 at 10:48 PM, Jacques Nadeau <jacques@dremio.com> wrote:

> I think we need to include DRILL-4467
> <https://issues.apache.org/jira/browse/DRILL-4467>. I think it is a one
> line patch and it provides unpredictable plans at a minimum but may also
> present invalid result. Still need to think through the second half. I've
> seen this plan instability in some of my recent test runs (even without
> Java 8) when running extended HBase tests.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Thu, Mar 3, 2016 at 10:02 PM, Parth Chandra <parthc@apache.org> wrote:
>
> > Updated list  (I'll follow up with the folks named here separately) -
> >
> > Committed for 1.6 -
> >
> > DRILL-4384 - Query profile is missing important information on WebUi -
> > Merged
> > DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged
> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) -
> > Waiting to be merged
> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). Waiting to
> > be merged.
> >
> > DRILL-4281/pr 400 (Drill should support inbound impersonation) (Jacques
> to
> > review)
> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> expose
> > their types within Calcite.) - Waiting for Aman to review. (Owners:
> Hsuan,
> > Jinfeng, Aman, Sudheesh)
> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > warnings about using the feature ) (Sudheesh to review.)
> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to
> > review)
> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh)
> >
> > Need review -
> > DRILL-4375/pr 402 (Fix the maven release profile)
> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > DRILL-4332/pr 389 (Make vector comparison order stable in test framework)
> > DRILL-4411/pr 381 (hash join over-memory condition)
> > DRILL-4387/pr 379 (GroupScan should not use star column)
> > DRILL-4184/pr 372 (support variable length decimal fields in parquet)
> > DRILL-4120 - dir0 does not work when the directory structure contains
> Avro
> > files - Partial patch available.
> > DRILL-4203/pr 341 (fix dates written into parquet files to conform to
> > parquet format spec)
> >
> > Not included (yet) -
> > DRILL-3149 - No patch available
> > DRILL-4441 - IN operator does not work with Avro reader - No patch
> > available
> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> > included in 1.6
> > DRILL-3623 - Limit 0 should avoid execution when querying a known schema.
> > (Need to add limitations of current impl). Intrusive change; should be
> > included at beginning of release cycle.
> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi) - Causes leak.
> >
> > Others -
> > DRILL-2517   - Already resolved.
> > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR
> > needs to be closed.
> >
> >
> > On Thu, Mar 3, 2016 at 9:44 PM, Parth Chandra <parthc@apache.org> wrote:
> >
> > > Right. My mistake. Thanks, Jacques, for reviewing.
> > >
> > > On Thu, Mar 3, 2016 at 9:08 PM, Zelaine Fong <zfong@maprtech.com>
> wrote:
> > >
> > >> DRILL-4281/pr 400 (Drill should support inbound impersonation)
> (Sudheesh
> > >> to
> > >> review)
> > >>
> > >> Sudheesh is the fixer of DRILL-4281, so I don't think he can be the
> > >> reviewer :).
> > >>
> > >> -- Zelaine
> > >>
> > >> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra <parthc@apache.org>
> > wrote:
> > >>
> > >> > Here's an updated list with names of reviewers added. If anyone else
> > is
> > >> > reviewing the open PRs please let me know. Some PRs have owners
> names
> > >> that
> > >> > I will follow up with.
> > >> > Jason, I've included your JIRA in the list.
> > >> >
> > >> >
> > >> > Committed for 1.6 -
> > >> >
> > >> > DRILL-4384 - Query profile is missing important information on
> WebUi -
> > >> > Merged
> > >> > DRILL-3488/pr 388 (Java 1.8 support) - Merged.
> > >> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged
> > >> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) -
> > Merged
> > >> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration)
> -
> > >> > Waiting to be merged
> > >> >
> > >> > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > >> (Sudheesh to
> > >> > review)
> > >> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> > >> expose
> > >> > their types within Calcite.) - Waiting for Aman to review. (Owners:
> > >> Hsuan,
> > >> > Jinfeng, Aman, Sudheesh)
> > >> > DRILL-4313/pr 396  (Improved client randomization. Update JIRA with
> > >> > warnings about using the feature ) (Sudheesh to review.)
> > >> > DRILL-4437 (and others)/pr 394 (Operator unit test framework).
> (Parth
> > to
> > >> > review)
> > >> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..)
> (Aman
> > to
> > >> > review)
> > >> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi)
> > >> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner:
> > Sudheesh)
> > >> >
> > >> > Need review -
> > >> > DRILL-4375/pr 402 (Fix the maven release profile)
> > >> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > >> > DRILL-4332/pr 389 (Make vector comparison order stable in test
> > >> framework)
> > >> > DRILL-4411/pr 381 (hash join over-memory condition)
> > >> > DRILL-4387/pr 379 (GroupScan should not use star column)
> > >> > DRILL-4184/pr 372 (support variable length decimal fields in
> parquet)
> > >> > DRILL-4120 - dir0 does not work when the directory structure
> contains
> > >> Avro
> > >> > files - Partial patch available.
> > >> > DRILL-4203/pr 341 (fix dates written into parquet files to conform
> to
> > >> > parquet format spec)
> > >> >
> > >> > Not included (yet) -
> > >> > DRILL-3149 - No patch available
> > >> > DRILL-4441 - IN operator does not work with Avro reader - No patch
> > >> > available
> > >> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not
> > >> > included in 1.6
> > >> > DRILL-3623 - Limit 0 should avoid execution when querying a known
> > >> schema.
> > >> > (Need to add limitations of current impl). Intrusive change; should
> be
> > >> > included at beginning of release cycle.
> > >> >
> > >> > Others -
> > >> > DRILL-2517   - Already resolved.
> > >> > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already
> merged.
> > PR
> > >> > needs to be closed.
> > >> >
> > >> >
> > >> >
> > >> > Thanks
> > >> >
> > >> > Parth
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse <
> > >> altekrusejason@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > I have updated the PR for the parquet date corruption issue that
> > >> didn't
> > >> > > make it into 1.5.
> > >> > >
> > >> > > https://github.com/apache/drill/pull/341
> > >> > > https://issues.apache.org/jira/browse/DRILL-4203
> > >> > >
> > >> > > If this can get reviewed, I think it would be good to get into
the
> > >> > release.
> > >> > > Any takers?
> > >> > >
> > >> > > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra <parthc@apache.org
> >
> > >> > wrote:
> > >> > >
> > >> > > > I've summarized the list of JIRs below.
> > >> > > > The first set of pull requests is under review (or have
some
> > >> reviewer
> > >> > > > assigned).
> > >> > > > The second set contains pull requests that need review.
We need
> > >> > > committers
> > >> > > > to review these. Please volunteer or these will not be able
to
> > make
> > >> it
> > >> > > into
> > >> > > > the release.
> > >> > > > The third set is Jira's that do not have a patch and/or
should
> not
> > >> be
> > >> > > > included because they require deeper scrutiny.
> > >> > > > I'm hoping we can finalize the list of PRs that can be reviewed
> by
> > >> > Friday
> > >> > > > morning and possibly *finalize the list of issues to be
included
> > by
> > >> > > Friday
> > >> > > > end of day* so please take some time to review the PRs.
> > >> > > > Also note that the QA team has offered to do sanity testing
once
> > we
> > >> > > decide
> > >> > > > on the final commit to be included, before the release candidate
> > is
> > >> > > rolled
> > >> > > > out, which helps with the release candidate moving forward
> > smoothly.
> > >> > > >
> > >> > > > Here's the list -
> > >> > > >
> > >> > > > *Committed for 1.6 -*
> > >> > > > DRILL-4281/pr 400 (Drill should support inbound impersonation)
> > >> > > > DRILL-4372/pr 377(?) (Drill Operators and Functions should
> > correctly
> > >> > > expose
> > >> > > > their types within Calcite.) - Waiting for Aman to review.
> > >> > > > DRILL-4313/pr 396  (Improved client randomization. Update
JIRA
> > with
> > >> > > > warnings about using the feature ) Sudheesh to review.
> > >> > > > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review
> > >> > > > DRILL-4437 (and others)/pr 394 (Operator unit test framework).
> > >> Parth to
> > >> > > > review
> > >> > > > DRILL-4384 - Query profile is missing important information
on
> > >> WebUi -
> > >> > > > Marked as resolved. Patch not applied?
> > >> > > >
> > >> > > > *Need review -*
> > >> > > > DRILL-4465/pr 401 (Simplify Calcite parsing & planning
> > integration)
> > >> > > > DRILL-4375/pr 402 (Fix the maven release profile)
> > >> > > > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite)
> > >> > > > DRILL-4332/pr 389 (Make vector comparison order stable in
test
> > >> > framework)
> > >> > > > DRILL-4449/pr 389 (Wrong results when metadata cache is
used..)
> > >> > > > DRILL-4416/pr 385 (quote path separator)
> > >> > > > DRILL-4411/pr 381 (hash join over-memory condition)
> > >> > > > DRILL-4410/pr 380 (listvector should initiatlize bits...)
> > >> > > > DRILL-4387/pr 379 (GroupScan should not use star column)
> > >> > > > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos,
etc)
> > >> > > > DRILL-4184/pr 372 (support variable length decimal fields
in
> > >> parquet)
> > >> > > > DRILL-4069/pr 352 Enable RPC thread offload by default
> > >> > > > DRILL-4120 - dir0 does not work when the directory structure
> > >> contains
> > >> > > Avro
> > >> > > > files - Partial patch available.
> > >> > > >
> > >> > > > *Not included (yet) - *
> > >> > > > DRILL-3149 - No patch available
> > >> > > > DRILL-4441 - IN operator does not work with Avro reader
- No
> patch
> > >> > > > available
> > >> > > > DRILL-3745/pr 399 - Hive char support - New feature - Needs
QA -
> > Not
> > >> > > > included in 1.6
> > >> > > > DRILL-3623 - Limit 0 should avoid execution when querying
a
> known
> > >> > schema.
> > >> > > > (Need to add limitations of current impl). Intrusive change;
> > should
> > >> be
> > >> > > > included at beginning of release cycle.
> > >> > > >
> > >> > > > *Others -*
> > >> > > > DRILL-2517   - Already resolved.
> > >> > > > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already
> > >> merged.
> > >> > PR
> > >> > > > needs to be closed.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Mar 2, 2016 at 3:11 PM, Vicky Markman <
> > >> vmarkman@maprtech.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > You are welcome, Jacques.
> > >> > > > >
> > >> > > > > Vick*y *:)
> > >> > > > >
> > >> > > > > On Wed, Mar 2, 2016 at 3:06 PM, Jacques Nadeau <
> > >> jacques@dremio.com>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > I just realized that we didn't merge the broken
profile
> patch
> > >> > (thanks
> > >> > > > > > Vicki). We should get it merged as well.
> > >> > > > > >
> > >> > > > > > DRILL-4384
> > >> > > > > >
> > >> > > > > > --
> > >> > > > > > Jacques Nadeau
> > >> > > > > > CTO and Co-Founder, Dremio
> > >> > > > > >
> > >> > > > > > On Wed, Mar 2, 2016 at 10:46 AM, Jason Altekruse
<
> > >> > > > > altekrusejason@gmail.com
> > >> > > > > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I should have merged this sooner but we will
need this
> patch
> > >> > that I
> > >> > > > had
> > >> > > > > > > applied to the 1.5 release branch. The change
is small and
> > >> fixes
> > >> > a
> > >> > > > > build
> > >> > > > > > > problem that only appears when running the
maven release
> > >> profile.
> > >> > > > > > >
> > >> > > > > > > https://github.com/apache/drill/pull/402
> > >> > > > > > >
> > >> > > > > > > On Wed, Mar 2, 2016 at 9:28 AM, Jinfeng Ni
<
> > >> > jinfengni99@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi John,
> > >> > > > > > > >
> > >> > > > > > > > I think patch for DRILL-2517 has been
merged to the
> apache
> > >> > master
> > >> > > > > > > > branch. Have you tried your query on
the latest master
> > >> branch?
> > >> > > > > > > >
> > >> > > > > > > > In DRILL-2517, I posted some performance
number for 117k
> > >> small
> > >> > > > > parquet
> > >> > > > > > > > files. The patch did show improvement.
> > >> > > > > > > >
> > >> > > > > > > > Before DRILL-3996 is resolved, for now
if your query
> > relies
> > >> on
> > >> > > > filter
> > >> > > > > > > > pushdown logic to push partitioning
filter first, then
> the
> > >> > patch
> > >> > > > for
> > >> > > > > > > > DRILL-2517 will not help.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Mar 2, 2016 at 4:23 AM, John
Omernik <
> > >> john@omernik.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > > > > I'd like to request drill-2517
be added as a bandaid
> for
> > >> the
> > >> > > > > planning
> > >> > > > > > > > > issues when there are lots of directories
of parquet
> > >> files.
> > >> > > >  This
> > >> > > > > > > issue
> > >> > > > > > > > is
> > >> > > > > > > > > really hurting drill adoption for
me in my org.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,  John
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Tuesday, March 1, 2016, Edmon
Begoli <
> > >> ebegoli@gmail.com>
> > >> > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> May I please ask to give this
issue the attention for
> > >> 1.6:
> > >> > > > > > > > >>
> > >> > > > > > >
> > >> > > >
> > >>
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/DRILL-3149
> > >> > > > > > > > >>
> > >> > > > > > > > >> I will try to suggest a patch.
Given my time
> > constraints
> > >> I
> > >> > > might
> > >> > > > > not
> > >> > > > > > > be
> > >> > > > > > > > >> able to submit a complete,
unit tested code but at I
> > >> least I
> > >> > > > will
> > >> > > > > > try
> > >> > > > > > > to
> > >> > > > > > > > >> submit a snippet that will
help with fixing it up (I
> > >> think
> > >> > we
> > >> > > > just
> > >> > > > > > > need
> > >> > > > > > > > to
> > >> > > > > > > > >> do a look ahead byte lookup
to ensure that it is not
> > >> \r\n.
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Tuesday, March 1, 2016,
Jacques Nadeau <
> > >> > jacques@dremio.com
> > >> > > > > > > > >> <javascript:;>> wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >> > It seems like a stretch
to include DRILL-3623 right
> > >> before
> > >> > > the
> > >> > > > > > > > release.
> > >> > > > > > > > >> > This is a pretty fundamental
change that seems like
> > it
> > >> > > should
> > >> > > > > soak
> > >> > > > > > > > for a
> > >> > > > > > > > >> > bit of time before we
release. If we want to
> include,
> > >> I'd
> > >> > > > > suggest
> > >> > > > > > > > that we
> > >> > > > > > > > >> > disable the functionality
by default and consider
> it
> > >> > > > > experimental.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > I'll propose a few other
patches for inclusion
> > shortly.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > --
> > >> > > > > > > > >> > Jacques Nadeau
> > >> > > > > > > > >> > CTO and Co-Founder, Dremio
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > On Tue, Mar 1, 2016 at
5:04 PM, Parth Chandra <
> > >> > > > > parthc@apache.org
> > >> > > > > > > > >> <javascript:;>
> > >> > > > > > > > >> > <javascript:;>>
wrote:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > > Hello everyone,
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >   It's time to start
looking into the 1.6
> release.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >   Can all the folks
working on open issues let me
> > >> know
> > >> > if
> > >> > > > > there
> > >> > > > > > > are
> > >> > > > > > > > any
> > >> > > > > > > > >> > > JIRAs you would like
to get into the release?
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >   I know of the following
-
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > *DRILL-4281 *- Drill
should support inbound
> > >> > impersonation.
> > >> > > > > Pull
> > >> > > > > > > > request
> > >> > > > > > > > >> > > expected today. C++
client to be tested.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > *DRILL-4372* - Drill
Operators and Functions
> should
> > >> > > > correctly
> > >> > > > > > > expose
> > >> > > > > > > > >> > their
> > >> > > > > > > > >> > > types within Calcite.
Waiting for review.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > *DRILL-3623* - Limit
0 should avoid execution
> when
> > >> > > querying
> > >> > > > a
> > >> > > > > > > known
> > >> > > > > > > > >> > schema.
> > >> > > > > > > > >> > > Pull request expected
today. Need to add
> > limitations
> > >> of
> > >> > > > > current
> > >> > > > > > > > impl to
> > >> > > > > > > > >> > the
> > >> > > > > > > > >> > > JIRA. Review needed.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > *DRILL-4313* - Improved
client randomization.
> > Update
> > >> > JIRA
> > >> > > > with
> > >> > > > > > > > warnings
> > >> > > > > > > > >> > > about using the feature.
Waiting for review.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > Thanks
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > Parth
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > > Sent from my iThing
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message