hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-5075) Potential infinite loop in updateMinSlots
Date Mon, 19 Jan 2009 00:59:59 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matei Zaharia updated HADOOP-5075:
----------------------------------

    Attachment: hadoop-5075-v2.patch

I think I found the cause for this now. The problem was with the weight normalization we did
in HADOOP-4789 to have equal shares per pools: The map weights were also being used for reduces.
The reason this caused a problem is because sometimes the map weight could be infinity if
the job had no runnable maps, and then we divided by this in updateMinSlots when calculating
guaranteed shares for reduces, leading to a 0 in both the ceil and the floor as I said. This
patch fixes that bug and also ensures that no infinities get introduced.

> Potential infinite loop in updateMinSlots
> -----------------------------------------
>
>                 Key: HADOOP-5075
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5075
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Priority: Blocker
>         Attachments: hadoop-5075-v2.patch, hadoop-5075.patch
>
>
> We ran into a problem at Facebook where the updateMinSlots loop in the scheduler was
repeating infinitely. This might happen if, due to rounding, we are unable to assign the last
few slots in a pool. This patch adds a break statement to ensure that the loop exists if it
hasn't managed to assign any slots.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message