giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avery Ching (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-302) Thread safety issue with sending partitions around
Date Fri, 17 Aug 2012 17:43:38 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436909#comment-13436909
] 

Avery Ching commented on GIRAPH-302:
------------------------------------

Thanks for the quick review Alessandro!

We can trigger a manual rebuild by logging in to hudson (upper right corner) and then clicking
"build now" (which I just did).  The error here was a strange ZK reconnect (we have this periodically).
 Rebuilding should fix it.

12/08/17 10:18:30 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid
0x139341759650003, likely server has closed socket, closing socket connection and attempting
reconnect
12/08/17 10:18:30 WARN zookeeper.ClientCnxn: Session 0x139341759650002 for server janus.apache.org/67.195.138.60:22181,
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcher.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
	at sun.nio.ch.IOUtil.read(IOUtil.java:200)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
	at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
12/08/17 10:18:30 ERROR zookeeper.ClientCnxn: Error while calling watcher 
java.lang.RuntimeException: process: Disconnected from ZooKeeper, cannot recover - WatchedEvent
state:Disconnected type:None path:null
	at org.apache.giraph.graph.BspService.process(BspService.java:1010)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
12/08/17 10:18:30 FATAL graph.GraphMapper: uncaughtException: OverrideExceptionHandler on
thread org.apache.giraph.graph.MasterThread, msg = barrierOnWorkerList: KeeperException -
Couldn't get children of /_hadoopBsp/job_local_0008/_applicationAttemptsDir/0/_superstepDir/5/_workerFinishedDir,
exiting...
java.lang.IllegalStateException: barrierOnWorkerList: KeeperException - Couldn't get children
of /_hadoopBsp/job_local_0008/_applicationAttemptsDir/0/_superstepDir/5/_workerFinishedDir
	at org.apache.giraph.graph.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1317)
	at org.apache.giraph.graph.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1507)
	at org.apache.giraph.graph.MasterThread.run(MasterThread.java:111)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /_hadoopBsp/job_local_0008/_applicationAttemptsDir/0/_superstepDir/5/_workerFinishedDir
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1243)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1271)
	at org.apache.giraph.zk.ZooKeeperExt.getChildrenExt(ZooKeeperExt.java:273)
	at org.apache.giraph.graph.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1311)
	... 2 more
                
> Thread safety issue with sending partitions around
> --------------------------------------------------
>
>                 Key: GIRAPH-302
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-302
>             Project: Giraph
>          Issue Type: Bug
>            Reporter: Avery Ching
>            Assignee: Avery Ching
>            Priority: Critical
>         Attachments: GIRAPH-302.patch
>
>
> When calling sendPartitionRequest(), we clear the vertex list afterward, making it a
race!
> I noticed this when I was running with 300 workers and the number of edges wasn't what
I expected.  Sometimes we get empty requests!
> After digging into the code I found the issue and have fixed it.
> Giraph Stats   Aggregate edges   99,971,220              0      99,971,220
> Superstep      11        0       11
> Current workers          300     0      300
> Last checkpointed superstep      0      0       0
> Current master task partition    0      0       0
> Sent messages  0    0            0
> Aggregate finished vertices      10,000,000     0       10,000,000
> Aggregate vertices               10,000,000     0       10,000,000
> This is wrong!
> Giraph Stats    Aggregate edges         100,000,000     0       100,000,000
> Superstep       11        0             11
> Last checkpointed superstep             0       0       0
> Current workers   300                   0       300
> Current master task partition           0       0       0
> Sent messages  0    0                   0
> Aggregate finished vertices             10,000,000      0       10,000,000
> Aggregate vertices                      10,000,000      0       10,000,000
> Fixed!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message