drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdel Hakim Deneche <adene...@maprtech.com>
Subject Re: anyone seen these errors on master ?
Date Fri, 14 Aug 2015 19:36:31 GMT
In addition to test*VectorReallocation errors I am seeing frequently on my
linux VM, today I saw the following error multiple times on my VM and once
on my Macbook:

Tests in error:
>   TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป IllegalState
> Failure...
> java.lang.IllegalStateException: Failure while closing accountor.
> Expected private and shared pools to be set to initial values.  However,
> one or more were not.  Stats are
> zone init allocated delta
> private 0 0 0
> shared 3221225472 3214166779 7058693.
> at
> org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)
> ~[classes/:na]
> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390)
> ~[classes/:na]
> at
> org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185)
> ~[classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)
> ~[classes/:na]
> at com.google.common.io.Closeables.close(Closeables.java:77)
> ~[guava-14.0.1.jar:na]
> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
> ~[guava-14.0.1.jar:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294)
> ~[classes/:na]
> at
> org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332)
> ~[classes/:na]


On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <chriswestin42@gmail.com>
wrote:

> Given that the difference is just
>
> > java.lang.Exception: Unexpected exception,
> > expected<org.apache.drill.exec.exception.OversizedAllocationException>
> but
> > was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
>
> The question of "what constitutes an oversized allocation?" comes to mind.
> Is this test fragile relative to being run in different environments?
> I haven't seen the test so how is the determination that something is
> oversized made? It seems like that criterion sometimes fails, and we get an
> OOM because whatever the request is is still very large.
>
>
> On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <hgunes@maprtech.com> wrote:
>
> > I don't seem to be able to re-prod this. Let me look at this and update
> you
> > all.
> >
> > On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche <
> > adeneche@maprtech.com>
> > wrote:
> >
> > > I didn't make any change, I'm running 2 forks (the default). I got
> those
> > > errors 3 times now, 2 on a linux VM and 1 on a linux physical node
> > >
> > > On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <hgunes@maprtech.com>
> > wrote:
> > >
> > > > Did you tighten your memory settings? How many forks are you running
> > > with?
> > > > I bet you are truly running out of memory while executing this
> > particular
> > > > test case.
> > > >
> > > > -H+
> > > >
> > > > On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam <
> skatkam@maprtech.com>
> > > > wrote:
> > > >
> > > > > b2bbd99 committed on July 6th introduced the test.
> > > > >
> > > > > > On Aug 5, 2015, at 10:21 AM, Jinfeng Ni <jinfengni99@gmail.com>
> > > wrote:
> > > > > >
> > > > > > In that case,  we probably need do binary search to figure out
> > which
> > > > > recent
> > > > > > patch is causing this problem.
> > > > > >
> > > > > > On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim Deneche <
> > > > > adeneche@maprtech.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Just got those errors on master too
> > > > > >>
> > > > > >> On Wed, Aug 5, 2015 at 9:07 AM, Abdel Hakim Deneche <
> > > > > adeneche@maprtech.com
> > > > > >>>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> I'm seeing those errors intermittently when building
my private
> > > > > branch, I
> > > > > >>> don't believe I made any change that would have caused
them.
> > Anyone
> > > > > seen
> > > > > >>> them too ?
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> > > > > >>>> Time elapsed: 2.043 sec  <<< ERROR!
> > > > > >>>> java.lang.Exception: Unexpected exception,
> > > > > >>>>
> > > >
> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> > > > > >> but
> > > > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> > > > > >>>> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> > > > > >>>> at
> > > > org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125)
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> > > > > >>>> Time elapsed: 0.436 sec  <<< ERROR!
> > > > > >>>> java.lang.Exception: Unexpected exception,
> > > > > >>>>
> > > >
> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> > > > > >> but
> > > > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> > > > > >>>> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85)
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> > > > > >>>> Time elapsed: 0.788 sec  <<< ERROR!
> > > > > >>>> java.lang.Exception: Unexpected exception,
> > > > > >>>>
> > > >
> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> > > > > >> but
> > > > > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> > > > > >>>> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> > > > > >>>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372)
> > > > > >>>> at
> > > > > >>>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142)
> > > > > >>>
> > > > > >>>
> > > > > >>> Thanks
> > > > > >>> --
> > > > > >>>
> > > > > >>> Abdelhakim Deneche
> > > > > >>>
> > > > > >>> Software Engineer
> > > > > >>>
> > > > > >>>  <http://www.mapr.com/>
> > > > > >>>
> > > > > >>>
> > > > > >>> Now Available - Free Hadoop On-Demand Training
> > > > > >>> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >>
> > > > > >> Abdelhakim Deneche
> > > > > >>
> > > > > >> Software Engineer
> > > > > >>
> > > > > >>  <http://www.mapr.com/>
> > > > > >>
> > > > > >>
> > > > > >> Now Available - Free Hadoop On-Demand Training
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > >   <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>



-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message