spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Kelly <jonathaka...@gmail.com>
Subject Re: spark on yarn wastes one box (or 1 GB on each box) for am container
Date Tue, 09 Feb 2016 23:28:36 GMT
Oh, sheesh, how silly of me. I copied and pasted that setting name without
even noticing the "mapreduce" in it. Yes, I guess that would mean that
Spark AMs are probably running even on TASK instances currently, which is
OK but not consistent with what we do for MapReduce. I'll make sure we
set spark.yarn.am.nodeLabelExpression appropriately in the next EMR release.

~ Jonathan

On Tue, Feb 9, 2016 at 1:30 PM Marcelo Vanzin <vanzin@cloudera.com> wrote:

> On Tue, Feb 9, 2016 at 12:16 PM, Jonathan Kelly <jonathakamzn@gmail.com>
> wrote:
> > And we do set yarn.app.mapreduce.am.labels=CORE
>
> That sounds very mapreduce-specific, so I doubt Spark (or anything
> non-MR) would honor it.
>
> --
> Marcelo
>

Mime
View raw message