drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: anyone seen these errors on master ?
Date Tue, 15 Sep 2015 17:21:05 GMT
I've seen sporadic failures with those tests. I'm wondering whether they
need to be redesigned. Limiting direct memory to 8gb shouldn't fail the
test.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Sep 15, 2015 at 9:39 AM, Abdel Hakim Deneche <adeneche@maprtech.com>
wrote:

> I found what was causing those errors on my VM. I exported a MAVEN_OPTS
> that limits the direct memory to 8GB and forget to unset it, this was
> causing the build to fail because it couldn't allocate more direct memory,
> for some reason it was always failing on those TestValueVector unit tests.
>
> On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com>
> wrote:
>
> > I am still seeing the TestValueVector unit tests fail on my linux machine
> > (2/3 of my builds)
> >
> > On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche <
> > adeneche@maprtech.com> wrote:
> >
> >> You can safely ignore the TestImpersonationMetadata error, it's most
> >> likely caused by my change
> >>
> >> On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche <
> >> adeneche@maprtech.com> wrote:
> >>
> >>> In addition to test*VectorReallocation errors I am seeing frequently on
> >>> my linux VM, today I saw the following error multiple times on my VM
> and
> >>> once on my Macbook:
> >>>
> >>> Tests in error:
> >>>>   TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป
> >>>> IllegalState Failure...
> >>>> java.lang.IllegalStateException: Failure while closing accountor.
> >>>> Expected private and shared pools to be set to initial values.
> However,
> >>>> one or more were not.  Stats are
> >>>> zone init allocated delta
> >>>> private 0 0 0
> >>>> shared 3221225472 3214166779 7058693.
> >>>> at
> >>>>
> org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)
> >>>> ~[classes/:na]
> >>>> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390)
> >>>> ~[classes/:na]
> >>>> at
> >>>>
> org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185)
> >>>> ~[classes/:na]
> >>>> at
> >>>>
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)
> >>>> ~[classes/:na]
> >>>> at com.google.common.io.Closeables.close(Closeables.java:77)
> >>>> ~[guava-14.0.1.jar:na]
> >>>> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
> >>>> ~[guava-14.0.1.jar:na]
> >>>> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294)
> >>>> ~[classes/:na]
> >>>> at
> >>>>
> org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332)
> >>>> ~[classes/:na]
> >>>
> >>>
> >>> On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <chriswestin42@gmail.com>
> >>> wrote:
> >>>
> >>>> Given that the difference is just
> >>>>
> >>>> > java.lang.Exception: Unexpected exception,
> >>>> >
> >>>>
> expected<org.apache.drill.exec.exception.OversizedAllocationException> but
> >>>> > was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> >>>>
> >>>> The question of "what constitutes an oversized allocation?" comes to
> >>>> mind.
> >>>> Is this test fragile relative to being run in different environments?
> >>>> I haven't seen the test so how is the determination that something is
> >>>> oversized made? It seems like that criterion sometimes fails, and we
> >>>> get an
> >>>> OOM because whatever the request is is still very large.
> >>>>
> >>>>
> >>>> On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <hgunes@maprtech.com>
> >>>> wrote:
> >>>>
> >>>> > I don't seem to be able to re-prod this. Let me look at this and
> >>>> update you
> >>>> > all.
> >>>> >
> >>>> > On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche <
> >>>> > adeneche@maprtech.com>
> >>>> > wrote:
> >>>> >
> >>>> > > I didn't make any change, I'm running 2 forks (the default).
I got
> >>>> those
> >>>> > > errors 3 times now, 2 on a linux VM and 1 on a linux physical
node
> >>>> > >
> >>>> > > On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <hgunes@maprtech.com
> >
> >>>> > wrote:
> >>>> > >
> >>>> > > > Did you tighten your memory settings? How many forks
are you
> >>>> running
> >>>> > > with?
> >>>> > > > I bet you are truly running out of memory while executing
this
> >>>> > particular
> >>>> > > > test case.
> >>>> > > >
> >>>> > > > -H+
> >>>> > > >
> >>>> > > > On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam <
> >>>> skatkam@maprtech.com>
> >>>> > > > wrote:
> >>>> > > >
> >>>> > > > > b2bbd99 committed on July 6th introduced the test.
> >>>> > > > >
> >>>> > > > > > On Aug 5, 2015, at 10:21 AM, Jinfeng Ni <
> >>>> jinfengni99@gmail.com>
> >>>> > > wrote:
> >>>> > > > > >
> >>>> > > > > > In that case,  we probably need do binary search
to figure
> out
> >>>> > which
> >>>> > > > > recent
> >>>> > > > > > patch is causing this problem.
> >>>> > > > > >
> >>>> > > > > > On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim
Deneche <
> >>>> > > > > adeneche@maprtech.com>
> >>>> > > > > > wrote:
> >>>> > > > > >
> >>>> > > > > >> Just got those errors on master too
> >>>> > > > > >>
> >>>> > > > > >> On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim
Deneche <
> >>>> > > > > adeneche@maprtech.com
> >>>> > > > > >>>
> >>>> > > > > >> wrote:
> >>>> > > > > >>
> >>>> > > > > >>> I'm seeing those errors intermittently
when building my
> >>>> private
> >>>> > > > > branch, I
> >>>> > > > > >>> don't believe I made any change that
would have caused
> them.
> >>>> > Anyone
> >>>> > > > > seen
> >>>> > > > > >>> them too ?
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> >>>> > > > > >>>> Time elapsed: 2.043 sec  <<<
ERROR!
> >>>> > > > > >>>> java.lang.Exception: Unexpected
exception,
> >>>> > > > > >>>>
> >>>> > > >
> >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> >>>> > > > > >> but
> >>>> > > > > >>>>
> >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> >>>> > > > > >>>> at
> >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> >>>> > > > > >>>> at
> java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> >>>> > > > > >>>> at
> >>>> > > >
> org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125)
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> >>>> > > > > >>>> Time elapsed: 0.436 sec  <<<
ERROR!
> >>>> > > > > >>>> java.lang.Exception: Unexpected
exception,
> >>>> > > > > >>>>
> >>>> > > >
> >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> >>>> > > > > >> but
> >>>> > > > > >>>>
> >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> >>>> > > > > >>>> at
> >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> >>>> > > > > >>>> at
> java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85)
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> >>>> > > > > >>>> Time elapsed: 0.788 sec  <<<
ERROR!
> >>>> > > > > >>>> java.lang.Exception: Unexpected
exception,
> >>>> > > > > >>>>
> >>>> > > >
> >>>> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> >>>> > > > > >> but
> >>>> > > > > >>>>
> >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> >>>> > > > > >>>> at
> >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> >>>> > > > > >>>> at
> java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372)
> >>>> > > > > >>>> at
> >>>> > > > > >>>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142)
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>> Thanks
> >>>> > > > > >>> --
> >>>> > > > > >>>
> >>>> > > > > >>> Abdelhakim Deneche
> >>>> > > > > >>>
> >>>> > > > > >>> Software Engineer
> >>>> > > > > >>>
> >>>> > > > > >>>  <http://www.mapr.com/>
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>> Now Available - Free Hadoop On-Demand
Training
> >>>> > > > > >>> <
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>> > > > > >>>
> >>>> > > > > >>>
> >>>> > > > > >>
> >>>> > > > > >>
> >>>> > > > > >>
> >>>> > > > > >> --
> >>>> > > > > >>
> >>>> > > > > >> Abdelhakim Deneche
> >>>> > > > > >>
> >>>> > > > > >> Software Engineer
> >>>> > > > > >>
> >>>> > > > > >>  <http://www.mapr.com/>
> >>>> > > > > >>
> >>>> > > > > >>
> >>>> > > > > >> Now Available - Free Hadoop On-Demand Training
> >>>> > > > > >> <
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>> > > > > >>>
> >>>> > > > > >>
> >>>> > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > >
> >>>> > > Abdelhakim Deneche
> >>>> > >
> >>>> > > Software Engineer
> >>>> > >
> >>>> > >   <http://www.mapr.com/>
> >>>> > >
> >>>> > >
> >>>> > > Now Available - Free Hadoop On-Demand Training
> >>>> > > <
> >>>> > >
> >>>> >
> >>>>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Abdelhakim Deneche
> >>>
> >>> Software Engineer
> >>>
> >>>   <http://www.mapr.com/>
> >>>
> >>>
> >>> Now Available - Free Hadoop On-Demand Training
> >>> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Abdelhakim Deneche
> >>
> >> Software Engineer
> >>
> >>   <http://www.mapr.com/>
> >>
> >>
> >> Now Available - Free Hadoop On-Demand Training
> >> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >>
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message