drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boaz Ben-Zvi <bben-...@mapr.com>
Subject Re: [DISCUSS] Drill 1.11.0 release
Date Tue, 18 Jul 2017 23:00:36 GMT
Hi Arina,

     The three Jira’s are ready (+1, squashed, rebased, etc.):

• DRILL-5616: Add memory checks, plus minor metrics changes
    (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a )
• DRILL-5665: Add the option planner.force_2phase_aggr to override small inputs
    (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e )
• DRILL-5669: Add a configurable option for minimum memory allocation to buffered ops
    (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 )

  Thanks,

         Boaz 

On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <arina.yelchiyeva@gmail.com> wrote:

    Hi Parth,
    
    no problem, I did not start preparing the release since there is one
    blocker has left:
    
    *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*
    
    
    Kind regards
    Arina
    
    On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <parthc@apache.org> wrote:
    
    > Hi Arina,
    >
    >   I just realized that I did not push the C++ client fix ( DRILL-5659) to
    > master. I have just done so.
    >
    >   Sorry if I messed things up for the release, but we should get this fix
    > in.
    >
    > Parth
    >
    > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <progers@mapr.com> wrote:
    >
    > > I looked into one NPE on CSV. Turns out it is already fixed by previous
    > > 1.11 PRs, so we’re good.
    > >
    > > There is another reported by a user on the user list, but I’ve not heard
    > > back about getting sample data. Let’s not hold the release for that.
    > >
    > > - Paul
    > >
    > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
    > > arina.yelchiyeva@gmail.com> wrote:
    > > >
    > > > *Blockers*
    > > > DRILL-5659 - Parth (done)
    > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
    > > >
    > > > *Possible blockers*
    > > > NPE on CSV - Paul can you please provide an update?
    > > >
    > > > Kind regards
    > > > Arina
    > > >
    > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
    > > > arina.yelchiyeva@gmail.com> wrote:
    > > >
    > > >> *Blockers*
    > > >> DRILL-5659 - problem is identified, Parth is working on the fix.
    > > >>
    > > >> *Not blockers*
    > > >> DRILL-5660 - Vitalii
    > > >> DRILL-5468 - Jinfeng
    > > >>
    > > >> *Possible blockers*
    > > >> DRILL-5669 - Boaz Ben-Zvi / Paul
    > > >> NPE on CSV - Paul
    > > >>
    > > >> Kind regards
    > > >> Arina
    > > >>
    > > >>
    > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <progers@mapr.com>
    > wrote:
    > > >>
    > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for
us
    > to
    > > >>> track down the incompatibility the issue internally, but production
    > > users
    > > >>> probably don’t attempt to use one build of Drill with metadata
files
    > > >>> created from another. We can worry about the issue later only if
a
    > user
    > > >>> actually runs into it.
    > > >>>
    > > >>> As suggested earlier, let’s just document this obscure case in
a
    > > release
    > > >>> note.
    > > >>>
    > > >>> - Paul
    > > >>>
    > > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jni@apache.org>
wrote:
    > > >>>>
    > > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet
metadata
    > > >>> cache
    > > >>>> file is involved.
    > > >>>>
    > > >>>> I might be wrong, but I thought that if someone is running
on
    > 1.10.0,
    > > >>> and
    > > >>>> he wants to go back to 1.0.0, where the first version of parquet
    > > >>> meatadata
    > > >>>> cache was introduced, can we guarantee that Drillbit would
work
    > > >>> correctly?
    > > >>>> I'm trying to understand what DRILL-5660 asks for is a new
    > > requirement,
    > > >>> or
    > > >>>> an requirement being there starting from Drill 1.0.0?
    > > >>>>
    > > >>>> Regression failures are normally regarded as release blocker.
    > > >>>>
    > > >>>>
    > > >>>>
    > > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <progers@mapr.com>
    > > wrote:
    > > >>>>
    > > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need
to do it at
    > > all,
    > > >>> so
    > > >>>>> that is a time savings. We could handle this via a release
note
    > item
    > > >>> that
    > > >>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible:
we
    > > do
    > > >>> not
    > > >>>>> support the ability to fall back to the old version. Everyone
OK
    > with
    > > >>> that?
    > > >>>>>
    > > >>>>> We are seeing some stress test regression failures that,
I’m told,
    > > will
    > > >>>>> show up as JIRA entries shortly.
    > > >>>>>
    > > >>>>> - Paul
    > > >>>>>
    > > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jni@apache.org>
wrote:
    > > >>>>>>
    > > >>>>>> I put some analysis in DRILL-5468.
    > > >>>>>>
    > > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
    > > >>>>>>
    > > >>>>>>
    > > >>>>>>
    > > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <
    > parthc@apache.org>
    > > >>>>> wrote:
    > > >>>>>>
    > > >>>>>>> Re DRILL-5634:
    > > >>>>>>> From the Apache commons web page [1] -
    > > >>>>>>>
    > > >>>>>>> Please note that Apache Commons Crypto doesn't
implement the
    > > >>>>> cryptographic
    > > >>>>>>> algorithm such as AES directly.* It wraps to Openssl
or JCE which
    > > >>>>> implement
    > > >>>>>>> the algorithms.*
    > > >>>>>>>
    > > >>>>>>>
    > > >>>>>>> Since we are cleared for using Openssl and JCE,
we are OK to
    > > include
    > > >>> the
    > > >>>>>>> AES code in the UDFs.
    > > >>>>>>>
    > > >>>>>>>
    > > >>>>>>> Parth
    > > >>>>>>>
    > > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
    > > >>>>>>>
    > > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva
<
    > > >>>>>>> arina.yelchiyeva@gmail.com> wrote:
    > > >>>>>>>
    > > >>>>>>>> During today hangouts we agreed on approximate
cut-off date - 14
    > > >>> July,
    > > >>>>>>> 2017
    > > >>>>>>>> (date may be shifted due to the blockers).
    > > >>>>>>>> I'll start preparing test RC on Thursday to
solve any release
    > > >>>>> preparation
    > > >>>>>>>> issues beforehand.
    > > >>>>>>>>
    > > >>>>>>>> *Blockers*
    > > >>>>>>>> DRILL-5660 - Vitalii
    > > >>>>>>>>
    > > >>>>>>>> *Possible blockers*
    > > >>>>>>>> DRILL-5468 - Jinfeng
    > > >>>>>>>> NPE on CSV source - Paul
    > > >>>>>>>> DRILL-5659 - Parth
    > > >>>>>>>>
    > > >>>>>>>> *Will be included if done before the cut-off
date*
    > > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
    > > >>>>>>>> DRILL-5634 - Parth will take a look at Apache
commons
    > > implementation
    > > >>>>> and
    > > >>>>>>>> update.
    > > >>>>>>>>
    > > >>>>>>>> Kind regards
    > > >>>>>>>> Arina
    > > >>>>>>>>
    > > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre
<
    > cgivre@gmail.com>
    > > >>>>> wrote:
    > > >>>>>>>>
    > > >>>>>>>>> Hi Arina,
    > > >>>>>>>>> I can’t make the hangout, but we have
a few options.
    > Basically,
    > > >>> since
    > > >>>>>>>> all
    > > >>>>>>>>> the hashing algorithms have no export restrictions,
we could:
    > > >>>>>>>>> 1.  Remove the AES encrypt/decrypt until
I can figure out what
    > we
    > > >>> have
    > > >>>>>>> to
    > > >>>>>>>>> do (and do it)
    > > >>>>>>>>> 2.  I can try to research it today or tomorrow
to figure out
    > > >>> exactly
    > > >>>>>>> what
    > > >>>>>>>>> we have to do to include AES.
    > > >>>>>>>>> 3.  Remove the entire package
    > > >>>>>>>>>
    > > >>>>>>>>> I’d like to see the hashing functions
at least included because
    > > >>> we’ve
    > > >>>>>>> had
    > > >>>>>>>>> a lot of requests for that functionality.
  I also don’t want
    > > this
    > > >>> to
    > > >>>>>>>> hold
    > > >>>>>>>>> things up, so please let me know your preference.
 Regardless,
    > > I’m
    > > >>>>>>> going
    > > >>>>>>>> to
    > > >>>>>>>>> start trying to figure out the restrictions
so we’ll know for
    > the
    > > >>>>>>> future.
    > > >>>>>>>>>
    > > >>>>>>>>> Best,
    > > >>>>>>>>> — C
    > > >>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva
<
    > > >>>>>>>> arina.yelchiyeva@gmail.com>
    > > >>>>>>>>> wrote:
    > > >>>>>>>>>>
    > > >>>>>>>>>> @Jinfeng,
    > > >>>>>>>>>> Let's discuss today on Drill hangout
final cut-off date.
    > > >>>>>>>>>>
    > > >>>>>>>>>> @Charles
    > > >>>>>>>>>> Since there were questions about AES
in DRILL-5634, should we
    > > >>> exclude
    > > >>>>>>>> it
    > > >>>>>>>>>> from the candidates for 1.11.0 release?
    > > >>>>>>>>>>
    > > >>>>>>>>>> @Boaz
    > > >>>>>>>>>> Do we have any ETA when DRILL-5616
and DRILL-5665 will pass
    > code
    > > >>>>>>>> review?
    > > >>>>>>>>>>
    > > >>>>>>>>>> Also I believe we should include DRILL-5660
where we need to
    > > bump
    > > >>> up
    > > >>>>>>>>>> parquet metadata version.
    > > >>>>>>>>>> @Vitalii
    > > >>>>>>>>>> Do you have any ETA for the fix?
    > > >>>>>>>>>>
    > > >>>>>>>>>> Kind regards
    > > >>>>>>>>>> Arina
    > > >>>>>>>>>>
    > > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz
Ben-Zvi <
    > > bben-zvi@mapr.com>
    > > >>>>>>>> wrote:
    > > >>>>>>>>>>
    > > >>>>>>>>>>> Hi Arina,
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>  Two PR have just been submitted,
relating to the Hash Aggr
    > > >>>>>>> Spill:
    > > >>>>>>>>>>> DRILL-5616 and DRILL-5665 .
    > > >>>>>>>>>>> Can these be considered for 1.11
as well ?
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>  Thanks,
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>            Boaz
    > > >>>>>>>>>>>
    > > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers"
<progers@mapr.com> wrote:
    > > >>>>>>>>>>>
    > > >>>>>>>>>>> DRILL-5601 is no longer a blocker
for 1.11; we’ll defer that
    > > >>>>>>> until
    > > >>>>>>>>>>> after the release to allow others
more time to work on other
    > > >>>>>>> features
    > > >>>>>>>>>>> rather than doing code reviews.
    > > >>>>>>>>>>>
    > > >>>>>>>>>>> - Paul
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM,
Jinfeng Ni <jni@apache.org>
    > > wrote:
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> Hi Arina,
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> It has been a while since we
discussed 1.11.0 release.  Any
    > > >>> update
    > > >>>>>>>>>>> on the
    > > >>>>>>>>>>>> new release plan?  Do we have
a cut-off date ?
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> In the past, I believe Drill
release is a time-bounded, not
    > > >>> feature
    > > >>>>>>>>>>>> bounded. If certain fixes/features
could not make it within
    > a
    > > >>>>>>> certain
    > > >>>>>>>>>>>> time-frame, we had better defer
them into future release,
    > > unless
    > > >>>>>>>>>>> those
    > > >>>>>>>>>>>> fixes/features are regarded
as blocking issues for a new
    > > >>> release.
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14
AM, Arina Yelchiyeva <
    > > >>>>>>>>>>> arina.yelchiyeva@gmail.com
    > > >>>>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>> Latest update:
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>> We wait for the following
Jiras before final cut off.
    > > >>>>>>>>>>>>> 1. DRILL-5601  - Paul
    > > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal
    > > >>>>>>>>>>>>> 3. DRILL-5634 - Charles
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>> Kind regards
    > > >>>>>>>>>>>>> Arina
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at
9:29 PM, Kunal Khatua <
    > > >>> kkhatua@mapr.com>
    > > >>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> Parth's already +1'd
the process with a minor update. So,
    > > >>> we're
    > > >>>>>>>>>>> good to
    > > >>>>>>>>>>>>> go
    > > >>>>>>>>>>>>>> on DRILL-5420. I'll
change the label on that to
    > > >>> ready-to-commit.
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> <http://www.mapr.com/>
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> ________________________________
    > > >>>>>>>>>>>>>> From: Parth Chandra
<parthc@apache.org>
    > > >>>>>>>>>>>>>> Sent: Thursday, June
29, 2017 9:44:39 AM
    > > >>>>>>>>>>>>>> To: dev@drill.apache.org
    > > >>>>>>>>>>>>>> Subject: Re: [DISCUSS]
Drill 1.11.0 release
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> I'm Ok with the pcap
format plugin if Pauls comments are
    > > also
    > > >>>>>>>>>>> addressed.
    > > >>>>>>>>>>>>> If
    > > >>>>>>>>>>>>>> the changes are done
before your cutoff deadline,we can
    > > merge
    > > >>> it
    > > >>>>>>>>>>> in.
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> On Thu, Jun 29, 2017
at 12:26 AM, Arina Yelchiyeva <
    > > >>>>>>>>>>>>>> arina.yelchiyeva@gmail.com>
wrote:
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> Hi all,
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> here is the list
of Jira we are waiting before making
    > final
    > > >>> cut
    > > >>>>>>>>>>> off:
    > > >>>>>>>>>>>>>>> 1. DRILL-5601 
- Paul
    > > >>>>>>>>>>>>>>> 2. DRILL-3640 
- Kunal
    > > >>>>>>>>>>>>>>> 3. DRILL-5420 -
Kunal
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> Please let me know
if there are any others Jiras we
    > should
    > > >>> wait
    > > >>>>>>>>>>> for.
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> @Charles,
    > > >>>>>>>>>>>>>>> you have mentioned
that you want to include some UDFs in
    > > >>> Drill
    > > >>>>>>>>>>> 1.11, do
    > > >>>>>>>>>>>>>> you
    > > >>>>>>>>>>>>>>> have any ETA for
the PR?
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> @Paul / Parth
    > > >>>>>>>>>>>>>>> I see there is
active CR for pcap format PR, do we want
    > to
    > > >>>>>>>>>>> include it
    > > >>>>>>>>>>>>> in
    > > >>>>>>>>>>>>>>> this release as
well?
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> Kind regards
    > > >>>>>>>>>>>>>>> Arina
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> On Thu, Jun 29,
2017 at 10:05 AM, Arina Yelchiyeva <
    > > >>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com>
wrote:
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>> Parth, thanks
a lot the instructions.
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>> On Wed, Jun
28, 2017 at 8:24 PM, Parth Chandra <
    > > >>>>>>>>>>> parthc@apache.org>
    > > >>>>>>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> FWIW, I
put release instructions culled from gists
    > > written
    > > >>> by
    > > >>>>>>>>>>>>> previous
    > > >>>>>>>>>>>>>>>>> release
managers into an updated gist here [1]
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> HTH
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> Parth
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
    > > >>>>>>>>>>>>> Release-Process
    > > >>>>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
    > > >>>>>>>>>>> Release-Process>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> On Tue,
Jun 27, 2017 at 5:24 PM, Paul Rogers <
    > > >>>>>>> progers@mapr.com>
    > > >>>>>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> Hi
Arina,
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> I have
two projects in progress: one is a candidate
    > for
    > > >>> 1.11,
    > > >>>>>>>>>>> the
    > > >>>>>>>>>>>>>>> other
    > > >>>>>>>>>>>>>>>>> is
    > > >>>>>>>>>>>>>>>>>> not.
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> The
“managed external sort” is the good candidate for
    > > >>> 1.11:
    > > >>>>>>>>>>> The PR
    > > >>>>>>>>>>>>>> for
    > > >>>>>>>>>>>>>>>>>> DRILL-5601
is being reviewed. Getting that in will
    > allow
    > > >>> us
    > > >>>>>>> to
    > > >>>>>>>>>>>>>> enable
    > > >>>>>>>>>>>>>>>>> the
    > > >>>>>>>>>>>>>>>>>> “new”
external sort by default.
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> The
other project is to address Drill’s memory
    > > >>> fragmentation
    > > >>>>>>>>>>> issue
    > > >>>>>>>>>>>>>>> with
    > > >>>>>>>>>>>>>>>>>> vectors
over 16 MB in size. Two PRs have been open
    > for a
    > > >>>>>>> while.
    > > >>>>>>>>>>>>> They
    > > >>>>>>>>>>>>>>> are
    > > >>>>>>>>>>>>>>>>>> the
first of several. Each takes a while to review.
    > So,
    > > >>> the
    > > >>>>>>>>>>>>> project
    > > >>>>>>>>>>>>>>> does
    > > >>>>>>>>>>>>>>>>>> not
fit 1.11. The work can go into the next release
    > > >>> instead.
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> Thanks,
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> - Paul
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>
    > > >>>>>>>
    > > >>>>>
    > > >>>>>
    > > >>>
    > > >>>
    > > >>
    > >
    > >
    >
    

Mime
View raw message