hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Using Yarn in end to end tests
Date Mon, 29 Sep 2014 16:43:54 GMT
I got the following message after clicking on the link:

must be logged in

Can you give login information ?

Cheers


On Mon, Sep 29, 2014 at 9:40 AM, Alex Newman <posix4e@gmail.com> wrote:

> I am currently developing tests that use a mini yarn cluster. Because it is
> running on circle-ci I need to use the absolute minimum amount of
> memory.
>
> I'm currently setting
>     conf.setFloat("yarn.
> nodemanager.vmem-pmem-ratio", 8.0f);
>     conf.setBoolean("mapreduce.map.speculative", false);
>     conf.setBoolean("mapreduce.reduce.speculative", false);
>     conf.setInt("yarn.scheduler.minimum-allocation-mb", 128);
>     conf.setInt("yarn.scheduler.maximum-allocation-mb", 256);
>     conf.setInt("yarn.nodemanager.resource.memory-mb", 256);
>     conf.setInt("mapreduce.map.memory.mb", 128);
>     conf.set("mapreduce.map.java.opts", "-Xmx128m");
>
>     conf.setInt("mapreduce.reduce.memory.mb", 128);
>     conf.set("mapreduce.reduce.java.opts", "-Xmx128m");
>     conf.setInt("mapreduce.task.io.sort.mb", 64);
>
>     conf.setInt("yarn.app.mapreduce.am.resource.mb", 128);
>     conf.set("yarn.app.mapreduce.am.command-opts", "-Xmx109m");
>
>     conf.setInt("yarn.scheduler.minimum-allocation-vcores", 1);
>     conf.setInt("yarn.scheduler.maximum-allocation-vcores", 1);
>     conf.setInt("yarn.nodemanager.resource.cpu-vcores", 1);
>     conf.setInt("mapreduce.map.cpu.vcore", 1);
>     conf.setInt("mapreduce.reduce.cpu.vcore", 1);
>
>     conf.setInt("mapreduce.tasktracker.map.tasks.maximum", 1);
>     conf.setInt("mapreduce.tasktracker.reduce.tasks.maximum", 1);
>
>     conf.setInt("yarn.scheduler.capacity.root.capacity",1);
>     conf.setInt("yarn.scheduler.capacity.maximum-applications", 1);
>
> conf.setInt("mapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob", 1);
>
> but I am still seeing many child tasks running
>
> https://circle-artifacts.com/gh/OhmData/hbase-public/314/artifacts/2/tmp/memory-usage.txt
>
> Any ideas on how to actually limit yarn to one or two children at a time?
>

Mime
View raw message