hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Szegedi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5686) DefaultContainerExecutor random working dir algorigthm skews results
Date Thu, 20 Oct 2016 16:48:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592333#comment-15592333
] 

Miklos Szegedi commented on YARN-5686:
--------------------------------------

Thank you, for the patch [~vrushalic]!

I think it would make sense to order these by the random number specified for better readability.
{code}
497	    assertEquals(1, executor.pickDirectory(100L, availableOnDisk));
498	    assertEquals(0, executor.pickDirectory(99L, availableOnDisk));
499	    assertEquals(1, executor.pickDirectory(101L, availableOnDisk));
500	    assertEquals(0, executor.pickDirectory(0L, availableOnDisk));
501	    assertEquals(1, executor.pickDirectory(199L, availableOnDisk));
{code}

> DefaultContainerExecutor random working dir algorigthm skews results
> --------------------------------------------------------------------
>
>                 Key: YARN-5686
>                 URL: https://issues.apache.org/jira/browse/YARN-5686
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Miklos Szegedi
>            Assignee: Vrushali C
>            Priority: Minor
>         Attachments: YARN-5686.001.patch
>
>
> {code}
> long randomPosition = RandomUtils.nextLong() % totalAvailable;
>     ...
>     while (randomPosition > availableOnDisk[dir]) {
>       randomPosition -= availableOnDisk[dir++];
>     }
> {code}
> The code above selects a disk based on the random number weighted by the free space on
each disk respectively. For example, if I have two disks with 100 bytes each, totalAvailable
is 200. The value of randomPosition will be 0..199. 0..99 should select the first disk, 100..199
should select the second disk inclusively. Random number 100 should select the second disk
to be fair but this is not the case right now.
> We need to use 
> {code}
> while (randomPosition >= availableOnDisk[dir])
> {code}
> instead of
> {code}
> while (randomPosition > availableOnDisk[dir])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message