spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Max Schmidt <...@datapath.io>
Subject Re: Where to set properties for the retainedJobs/Stages?
Date Fri, 01 Apr 2016 19:06:13 GMT
Yes but doc doesn't say any word for which variable the configs are 
valid, so do I have to set them for the history-server? The daemon? The 
workers?

And what if I use the java API instead of spark-submit for the jobs?

I guess that the spark-defaults.conf are obsolete for the java API?


Am 2016-04-01 18:58, schrieb Ted Yu:
> You can set them in spark-defaults.conf
>
> See 
> also https://spark.apache.org/docs/latest/configuration.html#spark-ui 
> [1]
>
> On Fri, Apr 1, 2016 at 8:26 AM, Max Schmidt <max@datapath.io> wrote:
>
>> Can somebody tell me the interaction between the properties:
>>
>> spark.ui.retainedJobs
>> spark.ui.retainedStages
>> spark.history.retainedApplications
>>
>> I know from the bugtracker, that the last one describes the number 
>> of
>> applications the history-server holds in memory.
>>
>> Can I set the properties in the spark-env.sh? And where?
>>
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
> Links:
> ------
> [1] https://spark.apache.org/docs/latest/configuration.html#spark-ui




---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message