giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tripti Singh <tri...@yahoo-inc.com>
Subject Re: YARN vs. MR1: is YARN a good idea ? Out-of-Core Graph ?
Date Tue, 28 Oct 2014 06:30:15 GMT
Hey Olivier,
I got onto some other stuff and didn¹t get time to try out anything
recently.

I was able to briefly resolve the out-of-memory by increasing the frame
decoder¹s frame length from 1GB to Integer.MAX_VALUE (search for
LengthFieldBasedFrameDecoder in NettyServer.java and change the first
parameter from 1024*1024*1024 to Integer.MAX_VALUE, in all the constructor
calls). I did this becoz I was getting the following stack trace:

exceptionCaught: Channel failed with remote address /10.216.185.26:52706
io.netty.handler.codec.TooLongFrameException: Adjusted frame length
exceeds 1073741824: 1647797607 - discarded
        at 
io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFr
ameDecoder.java:501)
        at 
io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthF
ieldBasedFrameDecoder.java:477)
        at 
io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBased
FrameDecoder.java:403)
        at 
io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBased
FrameDecoder.java:343)
        at 
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder
.java:226)
        at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecode
r.java:139)
        at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChan
nelHandlerContext.java:338)
        at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHand
lerContext.java:29)
        at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerCo
ntext.java:329)
        at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java
:36)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventE
xecutor.java:101)
        at java.lang.Thread.run(Thread.java:722)
exceptionCaught: Channel failed with remote address /10.216.198.32:37621
java.io.EOFException: fieldSize is too long! Length is 876099893, but
maximum is 20404
        at 
io.netty.buffer.ByteBufInputStream.checkAvailable(ByteBufInputStream.java:2
53)
        at 
io.netty.buffer.ByteBufInputStream.readFully(ByteBufInputStream.java:174)
        at 
io.netty.buffer.ByteBufInputStream.readFully(ByteBufInputStream.java:169)
        at 
org.apache.giraph.utils.WritableUtils.readExtendedDataOutput(WritableUtils.
java:379)
        at 
org.apache.giraph.utils.ByteArrayVertexIdData.readFields(ByteArrayVertexIdD
ata.java:52)
        at 
org.apache.giraph.utils.ByteArrayVertexIdMessages.readFields(ByteArrayVerte
xIdMessages.java:130)
        at 
org.apache.giraph.comm.requests.SendWorkerDataRequest.readFieldsRequest(Sen
dWorkerDataRequest.java:86)
        at 
org.apache.giraph.comm.requests.WritableRequest.readFields(WritableRequest.
java:118)
        at 
org.apache.giraph.utils.RequestUtils.decodeWritableRequest(RequestUtils.jav
a:52)
        at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDeco
der.java:89)
        at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChan
nelHandlerContext.java:338)
        at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHand
lerContext.java:29)
        at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerCo
ntext.java:329)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThread
EventExecutor.java:354)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventE
xecutor.java:101)
        at java.lang.Thread.run(Thread.java:722)
exceptionCaught: Channel failed with remote address /10.216.196.21:48709
java.lang.OutOfMemoryError: Java heap space
        at 
org.apache.giraph.utils.WritableUtils.readExtendedDataOutput(WritableUtils.
java:378)
        at 
org.apache.giraph.utils.ByteArrayVertexIdData.readFields(ByteArrayVertexIdD
ata.java:52)
        at 
org.apache.giraph.utils.ByteArrayVertexIdMessages.readFields(ByteArrayVerte
xIdMessages.java:130)
        at 
org.apache.giraph.comm.requests.SendWorkerDataRequest.readFieldsRequest(Sen
dWorkerDataRequest.java:86)
        at 
org.apache.giraph.comm.requests.WritableRequest.readFields(WritableRequest.
java:118)
        at 
org.apache.giraph.utils.RequestUtils.decodeWritableRequest(RequestUtils.jav
a:52)
        at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDeco
der.java:89)
        at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChan
nelHandlerContext.java:338)
        at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHand
lerContext.java:29)
        at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerCo
ntext.java:329)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThread
EventExecutor.java:354)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
        at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventE
xecutor.java:101)
        at java.lang.Thread.run(Thread.java:722)

If this works for you then that¹ll be great. But again, this failed for me
when input frame length was even higher than 2GB and I¹m yet to figure out
what is going on here.


Also, I did get a chance to try out the Out-of-core graph option and it
ran into a similar out-of-memory issue.
I will try to spend some time on this next week and will let u know if
there is a progress.

Thanks,
Tripti
Tech Yahoo




On 10/23/14, 3:32 AM, "Matthew Cornell" <matt@matthewcornell.org> wrote:

>Following up, does anyone have thoughts re: MR1 vs YARN performance?
>Thank you. -- matt
>
>On Wed, Oct 22, 2014 at 8:34 AM,  <olivier.varene@orange.com> wrote:
>> Hello Tripti,
>>
>> I bumped into this mail, I am experiencing out-of-memory errors on my
>>small cluster,
>> and, as out-of-core graph does not seem to work on giraph 1.1-SNAPSHOT,
>>I was wondering if you had any jira / patched already posted  to help
>>solve this issue ?
>>
>>
>> Thanks a lot
>>
>> regards
>>
>>
>> ------------------------------------
>> Olivier Varène
>> Big Data Referent
>> Orange - DSI France/Digital Factory
>> olivier.varene@orange.com
>> +33 4 97 46 29 94
>>
>>
>>
>>
>>
>>
>>
>> Le 10 oct. 2014 à 20:15, Tripti Singh <tripti@yahoo-inc.com> a écrit :
>>
>>> Hi Matthew,
>>> I would have been thrilled to give you numbers on this one but for me
>>>the Application is not scaling without the out-of-core option( which
>>>isn't working the way it was in previous version)
>>> I'm still figuring it out and can get back once it's resolved. I have
>>>patched a few things and will share them for people who might face
>>>similar issue. If u have a fix for scalability, do let me know
>>>
>>> Thanks,
>>> Tripti
>>>
>>> Sent from my iPhone
>>>
>>>> On 06-Oct-2014, at 9:22 pm, "Matthew Cornell"
>>>><matt@matthewcornell.org> wrote:
>>>>
>>>> Hi Folks. I don't think I paid enough attention to YARN vs. MR1 when I
>>>> built Giraph 1.0.0 for our system. How much better is Giraph on YARN?
>>>> Thank you.
>>>>
>>>> --
>>>> Matthew Cornell | matt@matthewcornell.org
>>
>>
>> 
>>_________________________________________________________________________
>>________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>>confidentielles ou privilegiees et ne doivent donc
>> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
>>recu ce message par erreur, veuillez le signaler
>> a l'expediteur et le detruire ainsi que les pieces jointes. Les
>>messages electroniques etant susceptibles d'alteration,
>> Orange decline toute responsabilite si ce message a ete altere, deforme
>>ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or privileged
>>information that may be protected by law;
>> they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and
>>delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have
>>been modified, changed or falsified.
>> Thank you.
>>
>
>
>
>-- 
>Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34
>Dickinson Street, Amherst MA 01002 | matthewcornell.org


Mime
View raw message