spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kankalapti omkar naidu <omkarnaid...@gmail.com>
Subject Rolling policy in Spark event logs for long living streaming applications
Date Fri, 01 Dec 2017 11:27:17 GMT
Dear spark developers,

I did not find rolling policy for spark event logs,for long living
streaming applications the event log is becoming huge with in few hours.
The only way I found is fixing the existing event logging spark
listener(EventLoggingListener),
but I believe there should be already a solution tackling this problem
because it is a frequently occurring problem when ever somebody runs
streaming applications for long time.

What would be the right way to fix this,whether any config settings are
available for this or writing our own scripts to cleanup the log directory
regularly based on time/size or we need to implement rolling policy feature?

Regards
Omkar

Mime
View raw message