drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Barclay <dbarc...@maprtech.com>
Subject Re: anyone seen these errors on master ?
Date Tue, 15 Sep 2015 20:12:49 GMT
I recently saw some failures that may be related.  Some test got a
different allocation exception than it expected.  I think that it was
expecting something like OversizeAllocationException but got an
OutOfMemoryError or OutOfMemoryException.

Daniel

Abdel Hakim Deneche wrote:
> I didn't go into the details of the failures, but the exception was caused
> by netty's not being able to allocate a new chunk of direct memory.
>
> On Tue, Sep 15, 2015 at 10:21 AM, Jacques Nadeau <jacques@dremio.com> wrote:
>
>> I've seen sporadic failures with those tests. I'm wondering whether they
>> need to be redesigned. Limiting direct memory to 8gb shouldn't fail the
>> test.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Tue, Sep 15, 2015 at 9:39 AM, Abdel Hakim Deneche <
>> adeneche@maprtech.com>
>> wrote:
>>
>>> I found what was causing those errors on my VM. I exported a MAVEN_OPTS
>>> that limits the direct memory to 8GB and forget to unset it, this was
>>> causing the build to fail because it couldn't allocate more direct
>> memory,
>>> for some reason it was always failing on those TestValueVector unit
>> tests.
>>>
>>> On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche <
>>> adeneche@maprtech.com>
>>> wrote:
>>>
>>>> I am still seeing the TestValueVector unit tests fail on my linux
>> machine
>>>> (2/3 of my builds)
>>>>
>>>> On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche <
>>>> adeneche@maprtech.com> wrote:
>>>>
>>>>> You can safely ignore the TestImpersonationMetadata error, it's most
>>>>> likely caused by my change
>>>>>
>>>>> On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche <
>>>>> adeneche@maprtech.com> wrote:
>>>>>
>>>>>> In addition to test*VectorReallocation errors I am seeing frequently
>> on
>>>>>> my linux VM, today I saw the following error multiple times on my
VM
>>> and
>>>>>> once on my Macbook:
>>>>>>
>>>>>> Tests in error:
>>>>>>>    TestImpersonationMetadata>BaseTestQuery.closeClient:239
ยป
>>>>>>> IllegalState Failure...
>>>>>>> java.lang.IllegalStateException: Failure while closing accountor.
>>>>>>> Expected private and shared pools to be set to initial values.
>>> However,
>>>>>>> one or more were not.  Stats are
>>>>>>> zone init allocated delta
>>>>>>> private 0 0 0
>>>>>>> shared 3221225472 3214166779 7058693.
>>>>>>> at
>>>>>>>
>>>
>> org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)
>>>>>>> ~[classes/:na]
>>>>>>> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390)
>>>>>>> ~[classes/:na]
>>>>>>> at
>>>>>>>
>>>
>> org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185)
>>>>>>> ~[classes/:na]
>>>>>>> at
>>>>>>>
>>>
>> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)
>>>>>>> ~[classes/:na]
>>>>>>> at com.google.common.io.Closeables.close(Closeables.java:77)
>>>>>>> ~[guava-14.0.1.jar:na]
>>>>>>> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
>>>>>>> ~[guava-14.0.1.jar:na]
>>>>>>> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294)
>>>>>>> ~[classes/:na]
>>>>>>> at
>>>>>>>
>>>
>> org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332)
>>>>>>> ~[classes/:na]
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <
>> chriswestin42@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Given that the difference is just
>>>>>>>
>>>>>>>> java.lang.Exception: Unexpected exception,
>>>>>>>>
>>>>>>>
>>> expected<org.apache.drill.exec.exception.OversizedAllocationException>
>> but
>>>>>>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
>>>>>>>
>>>>>>> The question of "what constitutes an oversized allocation?" comes
to
>>>>>>> mind.
>>>>>>> Is this test fragile relative to being run in different
>> environments?
>>>>>>> I haven't seen the test so how is the determination that something
>> is
>>>>>>> oversized made? It seems like that criterion sometimes fails,
and we
>>>>>>> get an
>>>>>>> OOM because whatever the request is is still very large.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <hgunes@maprtech.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't seem to be able to re-prod this. Let me look at this
and
>>>>>>> update you
>>>>>>>> all.
>>>>>>>>
>>>>>>>> On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche <
>>>>>>>> adeneche@maprtech.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I didn't make any change, I'm running 2 forks (the default).
I
>> got
>>>>>>> those
>>>>>>>>> errors 3 times now, 2 on a linux VM and 1 on a linux
physical
>> node
>>>>>>>>>
>>>>>>>>> On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <
>> hgunes@maprtech.com
>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Did you tighten your memory settings? How many forks
are you
>>>>>>> running
>>>>>>>>> with?
>>>>>>>>>> I bet you are truly running out of memory while executing
this
>>>>>>>> particular
>>>>>>>>>> test case.
>>>>>>>>>>
>>>>>>>>>> -H+
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam <
>>>>>>> skatkam@maprtech.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> b2bbd99 committed on July 6th introduced the
test.
>>>>>>>>>>>
>>>>>>>>>>>> On Aug 5, 2015, at 10:21 AM, Jinfeng Ni <
>>>>>>> jinfengni99@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> In that case,  we probably need do binary
search to figure
>>> out
>>>>>>>> which
>>>>>>>>>>> recent
>>>>>>>>>>>> patch is causing this problem.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 5, 2015 at 10:03 AM, Abdel Hakim
Deneche <
>>>>>>>>>>> adeneche@maprtech.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Just got those errors on master too
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 5, 2015 at 9:07 AM, Abdel
Hakim Deneche <
>>>>>>>>>>> adeneche@maprtech.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm seeing those errors intermittently
when building my
>>>>>>> private
>>>>>>>>>>> branch, I
>>>>>>>>>>>>>> don't believe I made any change that
would have caused
>>> them.
>>>>>>>> Anyone
>>>>>>>>>>> seen
>>>>>>>>>>>>>> them too ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
>>>>>>>>>>>>>>> Time elapsed: 2.043 sec  <<<
ERROR!
>>>>>>>>>>>>>>> java.lang.Exception: Unexpected
exception,
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> expected<org.apache.drill.exec.exception.OversizedAllocationException>
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>
>>>>>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
>>>>>>>>>>>>>>> at java.nio.Bits.reserveMemory(Bits.java:658)
>>>>>>>>>>>>>>> at
>>>>>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
>>>>>>>>>>>>>>> at
>>> java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
>>>>>>>>>>>>>>> at
>>>>>>>>>>
>>> org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
>>>>>>>>>>>>>>> Time elapsed: 0.436 sec  <<<
ERROR!
>>>>>>>>>>>>>>> java.lang.Exception: Unexpected
exception,
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> expected<org.apache.drill.exec.exception.OversizedAllocationException>
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>
>>>>>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
>>>>>>>>>>>>>>> at java.nio.Bits.reserveMemory(Bits.java:658)
>>>>>>>>>>>>>>> at
>>>>>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
>>>>>>>>>>>>>>> at
>>> java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
>>>>>>>>>>>>>>> Time elapsed: 0.788 sec  <<<
ERROR!
>>>>>>>>>>>>>>> java.lang.Exception: Unexpected
exception,
>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>> expected<org.apache.drill.exec.exception.OversizedAllocationException>
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>
>>>>>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
>>>>>>>>>>>>>>> at java.nio.Bits.reserveMemory(Bits.java:658)
>>>>>>>>>>>>>>> at
>>>>>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
>>>>>>>>>>>>>>> at
>>> java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Abdelhakim Deneche
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   <http://www.mapr.com/>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now Available - Free Hadoop On-Demand
Training
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Abdelhakim Deneche
>>>>>>>>>>>>>
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>
>>>>>>>>>>>>>   <http://www.mapr.com/>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now Available - Free Hadoop On-Demand
Training
>>>>>>>>>>>>> <
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Abdelhakim Deneche
>>>>>>>>>
>>>>>>>>> Software Engineer
>>>>>>>>>
>>>>>>>>>    <http://www.mapr.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>>>> <
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Abdelhakim Deneche
>>>>>>
>>>>>> Software Engineer
>>>>>>
>>>>>>    <http://www.mapr.com/>
>>>>>>
>>>>>>
>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>> <
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Abdelhakim Deneche
>>>>>
>>>>> Software Engineer
>>>>>
>>>>>    <http://www.mapr.com/>
>>>>>
>>>>>
>>>>> Now Available - Free Hadoop On-Demand Training
>>>>> <
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Abdelhakim Deneche
>>>>
>>>> Software Engineer
>>>>
>>>>    <http://www.mapr.com/>
>>>>
>>>>
>>>> Now Available - Free Hadoop On-Demand Training
>>>> <
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Abdelhakim Deneche
>>>
>>> Software Engineer
>>>
>>>    <http://www.mapr.com/>
>>>
>>>
>>> Now Available - Free Hadoop On-Demand Training
>>> <
>>>
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>
>>>
>>
>
>
>


-- 
Daniel Barclay
MapR Technologies

Mime
View raw message