Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
In-Reply-To: <5813ED55-2A5B-4108-8FD5-601EE9CF8E7E@mapr.com>
References: <CAHzp2tXXdd_BYbQr5F65RO+W=TYMOBkHBYpmNLTS1=dPg9ziFg@mail.gmail.com>
 <CAC1ju52P-cdbf1EBKMLfriLEgH3ZXwmCeSAfwHs+x0dUY86Xkw@mail.gmail.com>
 <64F4D25B-9BD8-4B94-9504-A03790279EF9@gmail.com> <CAC1ju50z9_G=OBnKfdhOUN5t42fj4MnyOnhO9WkDgbDf2Mz0iQ@mail.gmail.com>
 <7EACE15F-814E-4549-B677-9C60621AA3AA@mapr.com> <CAHzp2tVm26acK+JPO_+Ab_4H4qf4RG7oOMkB24jf-8DyMC2tAg@mail.gmail.com>
 <CAC1ju52KxY+0ORKF-PWg+cuckZud3c_+jGRDbdyBMqN_r5VcPg@mail.gmail.com>
 <CAC1ju50XMPoghUUs4jzv70CHS+AMLXhmropXp0CDOfcf4hStMA@mail.gmail.com>
 <CAHzp2tWXjjE+2TESY2ZW3v+P3A-AALHSun-2UE8k6raW-FQiJA@mail.gmail.com>
 <CY4PR16MB1688A9C11565D9EFBFC08EE5D2D30@CY4PR16MB1688.namprd16.prod.outlook.com>
 <CAC1ju51i6_YFhz5yCqSH_nC8Rwegv1Ci-jM92S=P9rMK2sjPBw@mail.gmail.com>
 <CAK55V=+o7_xEih4qD=ZKNhkF6LDCUvxzmMJDCTPsKxyaLAwP5Q@mail.gmail.com>
 <EB67C573-49CB-4C6C-9A02-F23BE8D4F44D@mapr.com> <6FF2B822-432E-4B8B-8B9F-9C726D379639@MAPR.COM>
 <CAC1ju50h1DgwU2_m=GT2Wjbp_=dLY3q489d7JoCYsxEKWep4Qg@mail.gmail.com>
 <7D412A8C-37FE-40FC-A680-D616CAD5CFD4@gmail.com> <CAC1ju52zQpHwaRM2H5E-QRZZM0_SV9mo2W5i4hMfh2bicqU5ZA@mail.gmail.com>
 <CAHzp2tVLro8=wxLxGECLX_vTOTbVBN0qy3zoZP8VR7fnJOEaEA@mail.gmail.com>
 <CAK55V=+t77x_m_XCdaG1PBMphHu9V+FHWBC0D1PGRwCij1iU+A@mail.gmail.com>
 <BE3AC524-DE0D-4201-86E1-9F6516BDB0C7@mapr.com> <CAK55V=K93PCqRNiLYYCQStfOODCfsEmCv5uNpeZ2J==iyZ1g-w@mail.gmail.com>
 <5813ED55-2A5B-4108-8FD5-601EE9CF8E7E@mapr.com>
From: Arina Yelchiyeva <arina.yelchiyeva@gmail.com>
Date: Thu, 13 Jul 2017 14:46:53 +0300
Message-ID: <CAC1ju53vm5i55GwQ1SWygGSqaBUW1qVEik=9XRc67G4rwwTMXQ@mail.gmail.com>
Subject: Re: [DISCUSS] Drill 1.11.0 release
To: dev@drill.apache.org
Content-Type: multipart/alternative; boundary="001a11c00d522d41210554317dea"
archived-at: Thu, 13 Jul 2017 11:47:08 -0000

--001a11c00d522d41210554317dea
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

*Blockers*
DRILL-5659 - problem is identified, Parth is working on the fix.

*Not blockers*
DRILL-5660 - Vitalii
DRILL-5468 - Jinfeng

*Possible blockers*
DRILL-5669 - Boaz Ben-Zvi / Paul
NPE on CSV - Paul

Kind regards
Arina


On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <progers@mapr.com> wrote:

> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to
> track down the incompatibility the issue internally, but production users
> probably don=E2=80=99t attempt to use one build of Drill with metadata fi=
les
> created from another. We can worry about the issue later only if a user
> actually runs into it.
>
> As suggested earlier, let=E2=80=99s just document this obscure case in a =
release
> note.
>
> - Paul
>
> > On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jni@apache.org> wrote:
> >
> > Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
> cache
> > file is involved.
> >
> > I might be wrong, but I thought that if someone is running on 1.10.0, a=
nd
> > he wants to go back to 1.0.0, where the first version of parquet
> meatadata
> > cache was introduced, can we guarantee that Drillbit would work
> correctly?
> > I'm trying to understand what DRILL-5660 asks for is a new requirement,
> or
> > an requirement being there starting from Drill 1.0.0?
> >
> > Regression failures are normally regarded as release blocker.
> >
> >
> >
> > On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <progers@mapr.com> wrote:
> >
> >> If we don=E2=80=99t do DRILL-5660 for 1.11 then we don=E2=80=99t need =
to do it at all,
> so
> >> that is a time savings. We could handle this via a release note item
> that
> >> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do
> not
> >> support the ability to fall back to the old version. Everyone OK with
> that?
> >>
> >> We are seeing some stress test regression failures that, I=E2=80=99m t=
old, will
> >> show up as JIRA entries shortly.
> >>
> >> - Paul
> >>
> >>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jni@apache.org> wrote:
> >>>
> >>> I put some analysis in DRILL-5468.
> >>>
> >>> Also, IMHO, DRILL-5660 is not a release blocker.
> >>>
> >>>
> >>>
> >>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <parthc@apache.org>
> >> wrote:
> >>>
> >>>> Re DRILL-5634:
> >>>> From the Apache commons web page [1] -
> >>>>
> >>>> Please note that Apache Commons Crypto doesn't implement the
> >> cryptographic
> >>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
> >> implement
> >>>> the algorithms.*
> >>>>
> >>>>
> >>>> Since we are cleared for using Openssl and JCE, we are OK to include
> the
> >>>> AES code in the UDFs.
> >>>>
> >>>>
> >>>> Parth
> >>>>
> >>>> [1] https://commons.apache.org/proper/commons-crypto/
> >>>>
> >>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> >>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>
> >>>>> During today hangouts we agreed on approximate cut-off date - 14
> July,
> >>>> 2017
> >>>>> (date may be shifted due to the blockers).
> >>>>> I'll start preparing test RC on Thursday to solve any release
> >> preparation
> >>>>> issues beforehand.
> >>>>>
> >>>>> *Blockers*
> >>>>> DRILL-5660 - Vitalii
> >>>>>
> >>>>> *Possible blockers*
> >>>>> DRILL-5468 - Jinfeng
> >>>>> NPE on CSV source - Paul
> >>>>> DRILL-5659 - Parth
> >>>>>
> >>>>> *Will be included if done before the cut-off date*
> >>>>> DRILL-5616 / DRILL-5665 - Boaz
> >>>>> DRILL-5634 - Parth will take a look at Apache commons implementatio=
n
> >> and
> >>>>> update.
> >>>>>
> >>>>> Kind regards
> >>>>> Arina
> >>>>>
> >>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cgivre@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> Hi Arina,
> >>>>>> I can=E2=80=99t make the hangout, but we have a few options.  Basi=
cally,
> since
> >>>>> all
> >>>>>> the hashing algorithms have no export restrictions, we could:
> >>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we
> have
> >>>> to
> >>>>>> do (and do it)
> >>>>>> 2.  I can try to research it today or tomorrow to figure out exact=
ly
> >>>> what
> >>>>>> we have to do to include AES.
> >>>>>> 3.  Remove the entire package
> >>>>>>
> >>>>>> I=E2=80=99d like to see the hashing functions at least included be=
cause
> we=E2=80=99ve
> >>>> had
> >>>>>> a lot of requests for that functionality.   I also don=E2=80=99t w=
ant this
> to
> >>>>> hold
> >>>>>> things up, so please let me know your preference.  Regardless, I=
=E2=80=99m
> >>>> going
> >>>>> to
> >>>>>> start trying to figure out the restrictions so we=E2=80=99ll know =
for the
> >>>> future.
> >>>>>>
> >>>>>> Best,
> >>>>>> =E2=80=94 C
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> >>>>> arina.yelchiyeva@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> @Jinfeng,
> >>>>>>> Let's discuss today on Drill hangout final cut-off date.
> >>>>>>>
> >>>>>>> @Charles
> >>>>>>> Since there were questions about AES in DRILL-5634, should we
> exclude
> >>>>> it
> >>>>>>> from the candidates for 1.11.0 release?
> >>>>>>>
> >>>>>>> @Boaz
> >>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
> >>>>> review?
> >>>>>>>
> >>>>>>> Also I believe we should include DRILL-5660 where we need to bump
> up
> >>>>>>> parquet metadata version.
> >>>>>>> @Vitalii
> >>>>>>> Do you have any ETA for the fix?
> >>>>>>>
> >>>>>>> Kind regards
> >>>>>>> Arina
> >>>>>>>
> >>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bben-zvi@mapr.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Arina,
> >>>>>>>>
> >>>>>>>>   Two PR have just been submitted, relating to the Hash Aggr
> >>>> Spill:
> >>>>>>>> DRILL-5616 and DRILL-5665 .
> >>>>>>>> Can these be considered for 1.11 as well ?
> >>>>>>>>
> >>>>>>>>   Thanks,
> >>>>>>>>
> >>>>>>>>             Boaz
> >>>>>>>>
> >>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <progers@mapr.com> wrote:
> >>>>>>>>
> >>>>>>>>  DRILL-5601 is no longer a blocker for 1.11; we=E2=80=99ll defer=
 that
> >>>> until
> >>>>>>>> after the release to allow others more time to work on other
> >>>> features
> >>>>>>>> rather than doing code reviews.
> >>>>>>>>
> >>>>>>>>  - Paul
> >>>>>>>>
> >>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jni@apache.org> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Arina,
> >>>>>>>>>
> >>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any
> update
> >>>>>>>> on the
> >>>>>>>>> new release plan?  Do we have a cut-off date ?
> >>>>>>>>>
> >>>>>>>>> In the past, I believe Drill release is a time-bounded, not
> feature
> >>>>>>>>> bounded. If certain fixes/features could not make it within a
> >>>> certain
> >>>>>>>>> time-frame, we had better defer them into future release, unles=
s
> >>>>>>>> those
> >>>>>>>>> fixes/features are regarded as blocking issues for a new releas=
e.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> >>>>>>>> arina.yelchiyeva@gmail.com
> >>>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Latest update:
> >>>>>>>>>>
> >>>>>>>>>> We wait for the following Jiras before final cut off.
> >>>>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>>>> 2. DRILL-5420 - Kunal
> >>>>>>>>>> 3. DRILL-5634 - Charles
> >>>>>>>>>>
> >>>>>>>>>> Kind regards
> >>>>>>>>>> Arina
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kkhatua@mapr.co=
m
> >
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Parth's already +1'd the process with a minor update. So, we'=
re
> >>>>>>>> good to
> >>>>>>>>>> go
> >>>>>>>>>>> on DRILL-5420. I'll change the label on that to
> ready-to-commit.
> >>>>>>>>>>>
> >>>>>>>>>>> <http://www.mapr.com/>
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________
> >>>>>>>>>>> From: Parth Chandra <parthc@apache.org>
> >>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> >>>>>>>>>>> To: dev@drill.apache.org
> >>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> >>>>>>>>>>>
> >>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
> >>>>>>>> addressed.
> >>>>>>>>>> If
> >>>>>>>>>>> the changes are done before your cutoff deadline,we can merge
> it
> >>>>>>>> in.
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> >>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> here is the list of Jira we are waiting before making final
> cut
> >>>>>>>> off:
> >>>>>>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>>>>>> 2. DRILL-3640  - Kunal
> >>>>>>>>>>>> 3. DRILL-5420 - Kunal
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please let me know if there are any others Jiras we should
> wait
> >>>>>>>> for.
> >>>>>>>>>>>>
> >>>>>>>>>>>> @Charles,
> >>>>>>>>>>>> you have mentioned that you want to include some UDFs in Dri=
ll
> >>>>>>>> 1.11, do
> >>>>>>>>>>> you
> >>>>>>>>>>>> have any ETA for the PR?
> >>>>>>>>>>>>
> >>>>>>>>>>>> @Paul / Parth
> >>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to
> >>>>>>>> include it
> >>>>>>>>>> in
> >>>>>>>>>>>> this release as well?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Kind regards
> >>>>>>>>>>>> Arina
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> >>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Parth, thanks a lot the instructions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> >>>>>>>> parthc@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> FWIW, I put release instructions culled from gists written
> by
> >>>>>>>>>> previous
> >>>>>>>>>>>>>> release managers into an updated gist here [1]
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> HTH
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Parth
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>>>>>> Release-Process
> >>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>>>> Release-Process>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
> >>>> progers@mapr.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Arina,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for
> 1.11,
> >>>>>>>> the
> >>>>>>>>>>>> other
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> not.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The =E2=80=9Cmanaged external sort=E2=80=9D is the good c=
andidate for 1.11:
> >>>>>>>> The PR
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow =
us
> >>>> to
> >>>>>>>>>>> enable
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> =E2=80=9Cnew=E2=80=9D external sort by default.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The other project is to address Drill=E2=80=99s memory
> fragmentation
> >>>>>>>> issue
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
> >>>> while.
> >>>>>>>>>> They
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>> the first of several. Each takes a while to review. So, t=
he
> >>>>>>>>>> project
> >>>>>>>>>>>> does
> >>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
> instead.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - Paul
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

--001a11c00d522d41210554317dea--