pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PIG-4922) Deadlock between SpillableMemoryManager and InternalSortedBag$SortedDataBagIterator
Date Mon, 06 Jun 2016 21:34:21 GMT

    [ https://issues.apache.org/jira/browse/PIG-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317274#comment-15317274
] 

Rohini Palaniswamy edited comment on PIG-4922 at 6/6/16 9:33 PM:
-----------------------------------------------------------------

Moved the sorting of the list that does getMemorySize outside synchronized(spillables) block
as we have already made the copy of the list to spillablesSR. 

getMemorySize has always been in synchronized(spillables). But seeing the issue for the first
time now. Most likely because in Tez, the input and output take more memory than mapreduce
and making it hit the spill code. 


was (Author: rohini):
Moved the sorting of the list that does getMemorySize outside synchronized(spillables) block
as we have already made the copy of the list to spillablesSR. 

> Deadlock between SpillableMemoryManager and InternalSortedBag$SortedDataBagIterator
> -----------------------------------------------------------------------------------
>
>                 Key: PIG-4922
>                 URL: https://issues.apache.org/jira/browse/PIG-4922
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.17.0, 0.16.1
>
>         Attachments: PIG-4922-1.patch
>
>
>   This one ran into a deadlock, when the data was really huge and InternalSortedBag was
reading spilled data from disk.
> {code}
> grpd = FOREACH (GROUP data BY $0){
>     sorted = ORDER data BY timestamp DESC;
>     latest = LIMIT sorted 1;
>     GENERATE latest;
> };
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message