hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Mitic (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2190) Provide a Windows container executor that can limit memory and CPU
Date Thu, 14 Aug 2014 06:05:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096610#comment-14096610
] 

Ivan Mitic commented on YARN-2190:
----------------------------------

Thanks Chuan for the patch. Looks great overall!

A few questions/suggestions below:
1. Where can I see that there are CPU/memory limits set on the job? ProcessExplorer?
2. Please make sure the code continues to compile/run on Win7/SDK. I am still on Server 2008R2
:) 
3. task.c: Can you init {{wchar_t *end}} to NULL? In the {{if}} check after wcstol, might
make sense to add {{end == NULL || *end !=...}}
4. task.c: ParseCommandLine: Given that you're passing pointers to variables on stack, you
could as well assert that {{memory}} and {{vcore}} are != NULL.
5. {code}
         OPTIONS: -c [cores] set virtual core limits on the job object.\n\

                  -m [memory] set the memory limit on the job object.\n\

{code}
Can you please specify the unit used for the memory/CPU limit?
6. task.c: {code}
    jcrci.CpuRate = vcores * (10000 / sysinfo.dwNumberOfProcessors);

{code}
Should we multiply first and then divide, to minimize precision loss?
7. Would you mind including a unittest for WindowsContainerExecutor? At this point it will
be a trivial test, but will likely grow over time. 
8. Just to confirm, by default, we will still use the DefaultContainerExecutor on Windows,
right? And users can configure the WindowsContainerExecutor if they want? This sounds good
until we develop better understand of how new limits behave in production. 

> Provide a Windows container executor that can limit memory and CPU
> ------------------------------------------------------------------
>
>                 Key: YARN-2190
>                 URL: https://issues.apache.org/jira/browse/YARN-2190
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>            Reporter: Chuan Liu
>            Assignee: Chuan Liu
>         Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, YARN-2190.2.patch
>
>
> Yarn default container executor on Windows does not set the resource limit on the containers
currently. The memory limit is enforced by a separate monitoring thread. The container implementation
on Windows uses Job Object right now. The latest Windows (8 or later) API allows CPU and memory
limits on the job objects. We want to create a Windows container executor that sets the limits
on job objects thus provides resource enforcement at OS level.
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message