giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claudio Martella (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-462) Multithreading breaks out-of-core graph
Date Tue, 22 Jan 2013 15:02:13 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559697#comment-13559697
] 

Claudio Martella commented on GIRAPH-462:
-----------------------------------------

This is exactly how I have implemented the LRU PartitionStore. Unfortunately, I have not been
able to test it in pseudo-distributed mode as I cannot make to run trunk on hadoop 1.0.4.
My PartitionStore is currently passing local tests and my ad-hoc tests for the PartitionStore.
I attach here the current patch.
                
> Multithreading breaks out-of-core graph
> ---------------------------------------
>
>                 Key: GIRAPH-462
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-462
>             Project: Giraph
>          Issue Type: Bug
>            Reporter: Alessandro Presta
>            Priority: Critical
>
> [~cmartella] pointed out this issue: when using multithreaded computation in conjunction
with out-of-core graph, we incur in a race condition. The compute threads share the same DiskBackedPartitionStore,
whose getPartition() method is not meant to be thread-safe. When two threads request two out-of-core
partitions concurrently, they both try to load it to the same slot.
> The result is that we can lose the reference to one of the two partitions (which will
not be written back to disk) and we can incur in a NullPointerException when both threads
are trying to offload the currently loaded partition to disk.
> I ran this test to confirm the issue:
> https://gist.github.com/4429628
> All tests pass except the one that uses both out-of-core graph and multiple compute threads.
> The error is the following:
> https://gist.github.com/4429650

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message