drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdel Hakim Deneche <adene...@maprtech.com>
Subject Re: anyone seen these errors on master ?
Date Tue, 15 Sep 2015 18:30:15 GMT
I didn't go into the details of the failures, but the exception was caused
by netty's not being able to allocate a new chunk of direct memory.

On Tue, Sep 15, 2015 at 10:21 AM, Jacques Nadeau <jacques@dremio.com> wrote:

> I've seen sporadic failures with those tests. I'm wondering whether they
> need to be redesigned. Limiting direct memory to 8gb shouldn't fail the
> test.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Sep 15, 2015 at 9:39 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com>
> wrote:
>
> > I found what was causing those errors on my VM. I exported a MAVEN_OPTS
> > that limits the direct memory to 8GB and forget to unset it, this was
> > causing the build to fail because it couldn't allocate more direct
> memory,
> > for some reason it was always failing on those TestValueVector unit
> tests.
> >
> > On Wed, Sep 2, 2015 at 10:10 AM, Abdel Hakim Deneche <
> > adeneche@maprtech.com>
> > wrote:
> >
> > > I am still seeing the TestValueVector unit tests fail on my linux
> machine
> > > (2/3 of my builds)
> > >
> > > On Fri, Aug 14, 2015 at 2:40 PM, Abdel Hakim Deneche <
> > > adeneche@maprtech.com> wrote:
> > >
> > >> You can safely ignore the TestImpersonationMetadata error, it's most
> > >> likely caused by my change
> > >>
> > >> On Fri, Aug 14, 2015 at 12:36 PM, Abdel Hakim Deneche <
> > >> adeneche@maprtech.com> wrote:
> > >>
> > >>> In addition to test*VectorReallocation errors I am seeing frequently
> on
> > >>> my linux VM, today I saw the following error multiple times on my VM
> > and
> > >>> once on my Macbook:
> > >>>
> > >>> Tests in error:
> > >>>>   TestImpersonationMetadata>BaseTestQuery.closeClient:239 ยป
> > >>>> IllegalState Failure...
> > >>>> java.lang.IllegalStateException: Failure while closing accountor.
> > >>>> Expected private and shared pools to be set to initial values.
> > However,
> > >>>> one or more were not.  Stats are
> > >>>> zone init allocated delta
> > >>>> private 0 0 0
> > >>>> shared 3221225472 3214166779 7058693.
> > >>>> at
> > >>>>
> >
> org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)
> > >>>> ~[classes/:na]
> > >>>> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:390)
> > >>>> ~[classes/:na]
> > >>>> at
> > >>>>
> >
> org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:185)
> > >>>> ~[classes/:na]
> > >>>> at
> > >>>>
> >
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)
> > >>>> ~[classes/:na]
> > >>>> at com.google.common.io.Closeables.close(Closeables.java:77)
> > >>>> ~[guava-14.0.1.jar:na]
> > >>>> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
> > >>>> ~[guava-14.0.1.jar:na]
> > >>>> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:294)
> > >>>> ~[classes/:na]
> > >>>> at
> > >>>>
> >
> org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:332)
> > >>>> ~[classes/:na]
> > >>>
> > >>>
> > >>> On Wed, Aug 5, 2015 at 2:52 PM, Chris Westin <
> chriswestin42@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Given that the difference is just
> > >>>>
> > >>>> > java.lang.Exception: Unexpected exception,
> > >>>> >
> > >>>>
> > expected<org.apache.drill.exec.exception.OversizedAllocationException>
> but
> > >>>> > was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> > >>>>
> > >>>> The question of "what constitutes an oversized allocation?" comes
to
> > >>>> mind.
> > >>>> Is this test fragile relative to being run in different
> environments?
> > >>>> I haven't seen the test so how is the determination that something
> is
> > >>>> oversized made? It seems like that criterion sometimes fails, and
we
> > >>>> get an
> > >>>> OOM because whatever the request is is still very large.
> > >>>>
> > >>>>
> > >>>> On Wed, Aug 5, 2015 at 2:26 PM, Hanifi Gunes <hgunes@maprtech.com>
> > >>>> wrote:
> > >>>>
> > >>>> > I don't seem to be able to re-prod this. Let me look at this
and
> > >>>> update you
> > >>>> > all.
> > >>>> >
> > >>>> > On Thu, Aug 6, 2015 at 12:03 AM, Abdel Hakim Deneche <
> > >>>> > adeneche@maprtech.com>
> > >>>> > wrote:
> > >>>> >
> > >>>> > > I didn't make any change, I'm running 2 forks (the default).
I
> got
> > >>>> those
> > >>>> > > errors 3 times now, 2 on a linux VM and 1 on a linux
physical
> node
> > >>>> > >
> > >>>> > > On Wed, Aug 5, 2015 at 1:03 PM, Hanifi Gunes <
> hgunes@maprtech.com
> > >
> > >>>> > wrote:
> > >>>> > >
> > >>>> > > > Did you tighten your memory settings? How many forks
are you
> > >>>> running
> > >>>> > > with?
> > >>>> > > > I bet you are truly running out of memory while
executing this
> > >>>> > particular
> > >>>> > > > test case.
> > >>>> > > >
> > >>>> > > > -H+
> > >>>> > > >
> > >>>> > > > On Wed, Aug 5, 2015 at 8:56 PM, Sudheesh Katkam
<
> > >>>> skatkam@maprtech.com>
> > >>>> > > > wrote:
> > >>>> > > >
> > >>>> > > > > b2bbd99 committed on July 6th introduced the
test.
> > >>>> > > > >
> > >>>> > > > > > On Aug 5, 2015, at 10:21 AM, Jinfeng Ni
<
> > >>>> jinfengni99@gmail.com>
> > >>>> > > wrote:
> > >>>> > > > > >
> > >>>> > > > > > In that case,  we probably need do binary
search to figure
> > out
> > >>>> > which
> > >>>> > > > > recent
> > >>>> > > > > > patch is causing this problem.
> > >>>> > > > > >
> > >>>> > > > > > On Wed, Aug 5, 2015 at 10:03 AM, Abdel
Hakim Deneche <
> > >>>> > > > > adeneche@maprtech.com>
> > >>>> > > > > > wrote:
> > >>>> > > > > >
> > >>>> > > > > >> Just got those errors on master too
> > >>>> > > > > >>
> > >>>> > > > > >> On Wed, Aug 5, 2015 at 9:07 AM, Abdel
Hakim Deneche <
> > >>>> > > > > adeneche@maprtech.com
> > >>>> > > > > >>>
> > >>>> > > > > >> wrote:
> > >>>> > > > > >>
> > >>>> > > > > >>> I'm seeing those errors intermittently
when building my
> > >>>> private
> > >>>> > > > > branch, I
> > >>>> > > > > >>> don't believe I made any change
that would have caused
> > them.
> > >>>> > Anyone
> > >>>> > > > > seen
> > >>>> > > > > >>> them too ?
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> testBitVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> > >>>> > > > > >>>> Time elapsed: 2.043 sec  <<<
ERROR!
> > >>>> > > > > >>>> java.lang.Exception: Unexpected
exception,
> > >>>> > > > > >>>>
> > >>>> > > >
> > >>>>
> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> > >>>> > > > > >> but
> > >>>> > > > > >>>>
> > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> > >>>> > > > > >>>> at
> > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> > >>>> > > > > >>>> at
> > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> > >>>> > > > > >>>> at
> > >>>> > > >
> > org.apache.drill.exec.vector.BitVector.reAlloc(BitVector.java:139)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.record.vector.TestValueVector.testBitVectorReallocation(TestValueVector.java:125)
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> testFixedVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> > >>>> > > > > >>>> Time elapsed: 0.436 sec  <<<
ERROR!
> > >>>> > > > > >>>> java.lang.Exception: Unexpected
exception,
> > >>>> > > > > >>>>
> > >>>> > > >
> > >>>>
> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> > >>>> > > > > >> but
> > >>>> > > > > >>>>
> > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> > >>>> > > > > >>>> at
> > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> > >>>> > > > > >>>> at
> > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:187)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:177)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:85)
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> testVariableVectorReallocation(org.apache.drill.exec.record.vector.TestValueVector)
> > >>>> > > > > >>>> Time elapsed: 0.788 sec  <<<
ERROR!
> > >>>> > > > > >>>> java.lang.Exception: Unexpected
exception,
> > >>>> > > > > >>>>
> > >>>> > > >
> > >>>>
> expected<org.apache.drill.exec.exception.OversizedAllocationException>
> > >>>> > > > > >> but
> > >>>> > > > > >>>>
> > >>>> was<org.apache.drill.exec.memory.OutOfMemoryRuntimeException>
> > >>>> > > > > >>>> at java.nio.Bits.reserveMemory(Bits.java:658)
> > >>>> > > > > >>>> at
> > >>>> java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> > >>>> > > > > >>>> at
> > java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.PooledByteBufAllocatorL.newDirectBuffer(PooledByteBufAllocatorL.java:130)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> io.netty.buffer.PooledByteBufAllocatorL.directBuffer(PooledByteBufAllocatorL.java:171)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:100)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.memory.TopLevelAllocator.buffer(TopLevelAllocator.java:116)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:372)
> > >>>> > > > > >>>> at
> > >>>> > > > > >>>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:142)
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>> Thanks
> > >>>> > > > > >>> --
> > >>>> > > > > >>>
> > >>>> > > > > >>> Abdelhakim Deneche
> > >>>> > > > > >>>
> > >>>> > > > > >>> Software Engineer
> > >>>> > > > > >>>
> > >>>> > > > > >>>  <http://www.mapr.com/>
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>> Now Available - Free Hadoop On-Demand
Training
> > >>>> > > > > >>> <
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >>>> > > > > >>>
> > >>>> > > > > >>>
> > >>>> > > > > >>
> > >>>> > > > > >>
> > >>>> > > > > >>
> > >>>> > > > > >> --
> > >>>> > > > > >>
> > >>>> > > > > >> Abdelhakim Deneche
> > >>>> > > > > >>
> > >>>> > > > > >> Software Engineer
> > >>>> > > > > >>
> > >>>> > > > > >>  <http://www.mapr.com/>
> > >>>> > > > > >>
> > >>>> > > > > >>
> > >>>> > > > > >> Now Available - Free Hadoop On-Demand
Training
> > >>>> > > > > >> <
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >>>> > > > > >>>
> > >>>> > > > > >>
> > >>>> > > > >
> > >>>> > > > >
> > >>>> > > >
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > --
> > >>>> > >
> > >>>> > > Abdelhakim Deneche
> > >>>> > >
> > >>>> > > Software Engineer
> > >>>> > >
> > >>>> > >   <http://www.mapr.com/>
> > >>>> > >
> > >>>> > >
> > >>>> > > Now Available - Free Hadoop On-Demand Training
> > >>>> > > <
> > >>>> > >
> > >>>> >
> > >>>>
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >>>> > > >
> > >>>> > >
> > >>>> >
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Abdelhakim Deneche
> > >>>
> > >>> Software Engineer
> > >>>
> > >>>   <http://www.mapr.com/>
> > >>>
> > >>>
> > >>> Now Available - Free Hadoop On-Demand Training
> > >>> <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Abdelhakim Deneche
> > >>
> > >> Software Engineer
> > >>
> > >>   <http://www.mapr.com/>
> > >>
> > >>
> > >> Now Available - Free Hadoop On-Demand Training
> > >> <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > >   <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>



-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message