drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: [VOTE] Release Apache Drill 1.3.0 (rc1)
Date Tue, 10 Nov 2015 02:14:04 GMT
Anybody want to profile what the impersonation tests are doing that are
causing so many problems with resources?

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Mon, Nov 9, 2015 at 5:24 PM, Hanifi Gunes <hgunes@maprtech.com> wrote:

> 1024/4096/2032195 <==> soft/hard/system
>
> On Mon, Nov 9, 2015 at 4:50 PM, Jacques Nadeau <jacques@dremio.com> wrote:
>
> > What about open files?
> > On Nov 9, 2015 4:20 PM, "Hanifi Gunes" <hgunes@maprtech.com> wrote:
> >
> > > Using the default parallelism, I get the same set of exceptions on
> > CentOS.
> > >
> > > Also max # of threads per process is the system default, 319548.
> > >
> > > On Mon, Nov 9, 2015 at 4:13 PM, Amit Hadke <amit.hadke@gmail.com>
> wrote:
> > >
> > > > Unfortunately, yes I changed them at OS level.
> > > >
> > > > I agree that making OS level changes to run tests is not something we
> > > > should advocate.
> > > > It was just an observation that tests seem to pass on linux with
> > default
> > > > settings but on mac behavior is irregular.
> > > >
> > > > ~ Amit.
> > > >
> > > >
> > > > On Mon, Nov 9, 2015 at 3:48 PM, Jason Altekruse <
> > > altekrusejason@gmail.com>
> > > > wrote:
> > > >
> > > > > Amit,
> > > > >
> > > > > Did you change these at the OS level? I would hope that we wouldn't
> > > have
> > > > to
> > > > > make people jump through such hoops to run the build, was there a
> > > thread
> > > > I
> > > > > missed where we discussed this something that we should be
> expecting
> > > > > everyone to do? I did see the threads about increasing the maven
> > > permgen
> > > > > and did so accordingly, but I think adjusting something like the
> > system
> > > > > ulimit is something we really should try to avoid.
> > > > >
> > > > > On Mon, Nov 9, 2015 at 3:44 PM, Abdel Hakim Deneche <
> > > > adeneche@maprtech.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I just tried with the default forkCount and got similar failures.
> > I'm
> > > > > > running the tests on a linux machine with the default maven
> options
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Nov 9, 2015 at 3:21 PM, Jacques Nadeau <
> jacques@dremio.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > So you're changing the forkCount from the default? What
happens
> > > when
> > > > > you
> > > > > > > run with the default?
> > > > > > >
> > > > > > > Also, did you guys run with extra permgen or the default?
Also,
> > can
> > > > you
> > > > > > > confirm what type of machine you are running on?
> > > > > > >
> > > > > > > Want to figure out why things are different.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Jacques
> > > > > > >
> > > > > > > --
> > > > > > > Jacques Nadeau
> > > > > > > CTO and Co-Founder, Dremio
> > > > > > >
> > > > > > > On Mon, Nov 9, 2015 at 2:23 PM, Abdel Hakim Deneche <
> > > > > > adeneche@maprtech.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Saw similar failures when running unit tests with
forkCount =
> > 1:
> > > > > > > >
> > > > > > > > Tests in error:
> > > > > > > > >
> > > > >  TestImpersonationQueries.sequenceFileChainedImpersonationWithView
> »
> > > > > > > > > UserRemote
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testMultiLevelImpersonationJoinEachSideReachesMaxUserHops:233->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213
> > > > > > > > > » Rpc
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testMultiLevelImpersonationExceedsMaxUserHops:219->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213
> > > > > > > > > » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.avroChainedImpersonationWithView:280->BaseTestImpersonation.createView:186->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213
> > > > > > > > > » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testDirectImpersonation_HasGroupReadPermissions:186->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213
> > > > > > > > > » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testDirectImpersonation_NoReadPermissions:196->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213
> > > > > > > > > » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testMultiLevelImpersonationEqualToMaxUserHops:210->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213
> > > > > > > > > » IllegalState
> > > > > > > > >   TestCountDownLatchInjection.latchInjected:146
NullPointer
> > > > > > > > >   TestMergingReceiver.testMultipleProvidersMixedSizes:115
»
> > > > > > > > > IllegalReferenceCount
> > > > > > > > > Tests run: 1482, Failures: 0, Errors: 9, Skipped:
118
> > > > > > > >
> > > > > > > >
> > > > > > > >  Unfortunately I'm changing my vote to -1
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Nov 9, 2015 at 2:21 PM, Hanifi Gunes <
> > > hgunes@maprtech.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > -1 (binding)
> > > > > > > > >
> > > > > > > > > 5/5 consecutive runs I get the following consistently
with
> > > > > forkCount
> > > > > > > set
> > > > > > > > to
> > > > > > > > > 1 on build.
> > > > > > > > >
> > > > > > > > > Tests in error:
> > > > > > > > >
> > > > >  TestImpersonationQueries.sequenceFileChainedImpersonationWithView
> »
> > > > > > > > > UserRemote
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testMultiLevelImpersonationJoinEachSideReachesMaxUserHops:233->BaseTestQuery.updateClient:222->BaseTestQuery.
> > > > > > > > >    updateClient:236->BaseTestQuery.updateClient:213
»
> > > > IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testMultiLevelImpersonationExceedsMaxUserHops:219->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:
> > > > > > > > >   236->BaseTestQuery.updateClient:213 » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.avroChainedImpersonationWithView:280->BaseTestImpersonation.createView:186->BaseTestQuery.updateClient:222-
> > > > > > > > >
> > > > > >BaseTestQuery.updateClient:236->BaseTestQuery.updateClient:213
> > > > > > »
> > > > > > > > > IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testDirectImpersonation_HasGroupReadPermissions:186->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:
> > > > > > > > > 236->BaseTestQuery.updateClient:213 » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testDirectImpersonation_NoReadPermissions:196->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:236-
> > > > > > > > >   >BaseTestQuery.updateClient:213 » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> TestImpersonationQueries.testMultiLevelImpersonationEqualToMaxUserHops:210->BaseTestQuery.updateClient:222->BaseTestQuery.updateClient:
> > > > > > > > >   236->BaseTestQuery.updateClient:213 » IllegalState
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Nov 9, 2015 at 2:19 PM, Stefán Baxter
<
> > > > > > > stefan@activitystream.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I'm not a committer and not voting but I
want to point
> out
> > > that
> > > > > the
> > > > > > > > UDFs
> > > > > > > > > > that we are using and run with 1.1 and 1.2
do not, for
> some
> > > > > reason,
> > > > > > > run
> > > > > > > > > > with 1.3.
> > > > > > > > > >
> > > > > > > > > > I'm inclined to insist that it's something
we are doing
> > wrong
> > > > but
> > > > > > if
> > > > > > > > this
> > > > > > > > > > counts as a test then it's failing.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >  -Stefan
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 9, 2015 at 10:09 PM, Norris
Lee <
> > > norrisl@simba.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 Non-binding.
> > > > > > > > > > >
> > > > > > > > > > > Built from source on Linux, tested
ODBC against various
> > > data
> > > > > > > sources
> > > > > > > > > > > (Hive, DFS-csv, tsv, parquet, json)
> > > > > > > > > > >
> > > > > > > > > > > Norris
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Abdel Hakim Deneche [mailto:
> adeneche@maprtech.com]
> > > > > > > > > > > Sent: Monday, November 09, 2015 10:16
AM
> > > > > > > > > > > To: dev@drill.apache.org
> > > > > > > > > > > Subject: Re: [VOTE] Release Apache
Drill 1.3.0 (rc1)
> > > > > > > > > > >
> > > > > > > > > > > +1
> > > > > > > > > > >
> > > > > > > > > > > Checked checksum files and gpg signature
for all 3
> files.
> > > > Built
> > > > > > > from
> > > > > > > > > > > source, deployed on a 4 nodes cluster
then run several
> > > window
> > > > > > > > function
> > > > > > > > > > > queries on TPCDS SF100. Looks great.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 9, 2015 at 8:14 AM, Jacques
Nadeau <
> > > > > > jacques@dremio.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I've seen another instance of
DRILL-4041. Since it is
> > > > > > exceedingly
> > > > > > > > > > > > rare, I don't think it is a release
stopper. I will
> > spend
> > > > > some
> > > > > > > more
> > > > > > > > > > > > time reviewing it.
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Jacques Nadeau
> > > > > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 8, 2015 at 7:34 PM,
Amit Hadke <
> > > > > > amit.hadke@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1, non-binding
> > > > > > > > > > > > > Downloaded tarball, mvn clean
install Ran join
> > > queries,,
> > > > > was
> > > > > > > able
> > > > > > > > > to
> > > > > > > > > > > > > run queries on json, parquet
and sequence files.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Nov 8, 2015 at 10:01
AM, Abhijit Pol <
> > > > > > > abhijit@dremio.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > resending the vote with
details
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1, non-binding
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ran maven tests, installed
on Mac, ran drill in
> > > > embedded
> > > > > > > mode,
> > > > > > > > > > > > > > verified
> > > > > > > > > > > > > Web
> > > > > > > > > > > > > > UI for CSV header config
changes, ran few queries
> > on
> > > > CSV
> > > > > > > files
> > > > > > > > > > > > > > with and without header,
ran few queries against
> > > large
> > > > > CSV
> > > > > > > > files
> > > > > > > > > > > > > > with header in
> > > > > > > > > > > > > HDFS
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Nov 8, 2015
at 9:07 AM, Hsuan Yi Chu <
> > > > > > > > > hyichu@maprtech.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > mvn clean install
on mac and linux
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > tried a few queries
on distributed mode.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Things looked fine
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Nov 8,
2015 at 8:16 AM, Aman Sinha <
> > > > > > > > > asinha@maprtech.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1.
> > > > > > > > > > > > > > > > Downloaded
the binary tar ball and installed
> on
> > > > Mac.
> > > > > > Ran
> > > > > > > > > > > > > > > > Drill in
> > > > > > > > > > > > > > > embedded
> > > > > > > > > > > > > > > > mode.  Tested
a few join, aggregation and
> limit
> > > > > queries
> > > > > > > > > > > > > > > > against
> > > > > > > > > > > > > parquet
> > > > > > > > > > > > > > > > data with
and without metadata cache.  Tested
> > > > > > > cancellation
> > > > > > > > > for
> > > > > > > > > > > > couple
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > queries. 
Looked at query profiles on the
> > WebUI.
> > > > > >  Looks
> > > > > > > > > good.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sat, Nov
7, 2015 at 11:53 PM, Abhijit Pol
> > > > > > > > > > > > > > > > <abhijit@dremio.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sat,
Nov 7, 2015 at 9:41 PM, andrew <
> > > > > > > > andrew@primer.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
+1 (non-binding, non-comitter)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> On Nov 6, 2015, at 10:15 PM, Jacques
> > > Nadeau <
> > > > > > > > > > > > > jacques@dremio.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> Hey Everybody,
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> I'd like to propose a new release
> > candidate
> > > > of
> > > > > > > Apache
> > > > > > > > > > > > > > > > > >
> Drill,
> > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > >
> 1.3.0.  This is the second release
> > > candidate
> > > > > > (rc1).
> > > > > > > > > > > > > > > > > >
> This
> > > > > > > > > > > > > > addresses
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > >
> issues identified in the first release
> > > > > candidate
> > > > > > > > > > > > > > > > > >
> including
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > test
> > > > > > > > > > > > > > > > > >
> threading issues, Parquet dictionary
> read
> > > > issue
> > > > > > > with
> > > > > > > > > > > > > > > > > >
> binary
> > > > > > > > > > > > > > values,
> > > > > > > > > > > > > > > > > Joda
> > > > > > > > > > > > > > > > > >
> upgrade to avoid a concurrency bug and
> a
> > > > couple
> > > > > > > other
> > > > > > > > > > > > > > > > > >
> small
> > > > > > > > > > > > > > things.
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> The tarball artifacts are hosted at [2]
> > and
> > > > the
> > > > > > > maven
> > > > > > > > > > > > artifacts
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > >
hosted
> > > > > > > > > > > > > > > > > >
> at [3]. This release candidate is based
> > on
> > > > > commit
> > > > > > > > > > > > > > > > > >
>
> f17ebd2fbf2bf5ba201a635f5b6ec3615afe2305
> > > > > located
> > > > > > at
> > > > > > > > > [4].
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> The vote will be open for 72 hours
> ending
> > > at
> > > > > 10PM
> > > > > > > > > > > > > > > > > >
> Pacific,
> > > > > > > > > > > > > > November
> > > > > > > > > > > > > > > > 9,
> > > > > > > > > > > > > > > > > >
2015.
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> [ ] +1
> > > > > > > > > > > > > > > > > >
> [ ] +0
> > > > > > > > > > > > > > > > > >
> [ ] -1
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> thanks,
> > > > > > > > > > > > > > > > > >
> Jacques
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> [1]
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313
> > > > > > > > > > > > 820&version=12332946[2]
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > http://people.apache.org/~jacques/apache-drill-1.3.0.rc1
> > > > > > > > > > > > > > > > > >
> /
> > > > > > > > > > > > > > > > > >
> [3]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://repository.apache.org/content/repositories/orgapachedrill-1014
> > > > > > > > > > > > /
> > > > > > > > > > > > > > > > > >
> [4]
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > https://github.com/jacques-n/drill/tree/drill-1.3.0-rc1
> > > > > > > > > > > > > > > > > >
>
> > > > > > > > > > > > > > > > > >
> --
> > > > > > > > > > > > > > > > > >
> Jacques Nadeau
> > > > > > > > > > > > > > > > > >
> CTO and Co-Founder, Dremio
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > Abdelhakim Deneche
> > > > > > > > > > >
> > > > > > > > > > > Software Engineer
> > > > > > > > > > >
> > > > > > > > > > >   <http://www.mapr.com/>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Now Available - Free Hadoop On-Demand
Training <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Abdelhakim Deneche
> > > > > > > >
> > > > > > > > Software Engineer
> > > > > > > >
> > > > > > > >   <http://www.mapr.com/>
> > > > > > > >
> > > > > > > >
> > > > > > > > Now Available - Free Hadoop On-Demand Training
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Abdelhakim Deneche
> > > > > >
> > > > > > Software Engineer
> > > > > >
> > > > > >   <http://www.mapr.com/>
> > > > > >
> > > > > >
> > > > > > Now Available - Free Hadoop On-Demand Training
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message