spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghavendra Pandey <raghavendra.pan...@gmail.com>
Subject Re: ALS.trainImplicit running out of mem when using higher rank
Date Mon, 19 Jan 2015 02:27:16 GMT
If you are running spark in local mode, executor parameters are not used as
there is no executor. You should try to set corresponding driver parameter
to effect it.

On Mon, Jan 19, 2015, 00:21 Sean Owen <sowen@cloudera.com> wrote:

> OK. Are you sure the executor has the memory you think? "-Xmx24g" in
> its command line? It may be that for some reason your job is reserving
> an exceptionally large amount of non-heap memory. I am not sure that's
> to be expected with the ALS job though. Even if the settings work,
> considering using the explicit command line configuration.
>
> On Sat, Jan 17, 2015 at 12:49 PM, Antony Mayi <antonymayi@yahoo.com>
> wrote:
> > the values are for sure applied as expected - confirmed using the spark
> UI
> > environment page...
> >
> > it comes from my defaults configured using
> > 'spark.yarn.executor.memoryOverhead=8192' (yes, now increased even
> more) in
> > /etc/spark/conf/spark-defaults.conf and 'export
> SPARK_EXECUTOR_MEMORY=24G'
> > in /etc/spark/conf/spark-env.sh
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message