giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-300) Improve netty reliability with retrying failed connections, tracking requests, thread-safe hash partitioning
Date Tue, 14 Aug 2012 17:57:38 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434328#comment-13434328
] 

Eli Reisman commented on GIRAPH-300:
------------------------------------

Can't wait to try this out, fantastic!

We have not found any hard limits on # of workers == Netty fail here, but when any one worker
for any reason starts getting Netty buffers backed up, thats when it will happen. Sometimes
its during temp partition shuffle on INPUT_SUPERSTEP, sometimes (more rare) its during computation
in a message intensive algorithm.

I got one run in this morning to try out some code, hope to get more in over the next few
days and this just made my short list. Review forthcoming, looking forward to it!

                
> Improve netty reliability with retrying failed connections, tracking requests, thread-safe
hash partitioning
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: GIRAPH-300
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-300
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Avery Ching
>            Assignee: Avery Ching
>         Attachments: GIRAPH-300.patch
>
>
> * Upgrade to the most recent stable version of Netty (3.5.3.Final)
> * Try multiple connection attempts up to n failures
> * Track requests throughout the system by keeping track of the request id and then matching
the request id to the response (minor refactoring of WritableRequest to make requests simpler
and support the request id)
> * Improved handling of netty exceptions by dumping the exception stack to help debug
failures
> * Fixes bug in HashWorkerPartitioner by making partitionList thread-safe (this causes
divide by zero exceptions in real life)
> Currently, netty connection failures causes issues with more than 75 workers in my setup.
 This allows us to reach over 200+ in a reasonably reliable network that doesn't kill connections.
> This code passes the local Hadoop regressions and the single node Hadoop instance regressions.
 It also succeeded on large runs (200+ workers) on a real Hadoop cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message