samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navina Ramesh <nram...@linkedin.com.INVALID>
Subject Re: Keep samza container logs from being deleted
Date Mon, 03 Aug 2015 22:06:16 GMT
Hi Chen,
You can set the yarn.nodemanage.delete.debug-delay-sec config in
yarn-site.xml, in order to retain the logs after the job crashes. Refer
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml


(This was intended for debug purpose).

If you need to retain log for longer period of time for purposes other than
debugging, you should look into yarn's log aggregation features.
Alternatively, you can use Samza's StreamAppender to produce the log data
to a stream. You can then publish it to ELK for further lookup. It really
depends on your requirements.
Please note that the StreamAppender has a bug in the master branch that is
pending resolution in https://issues.apache.org/jira/browse/SAMZA-723

Cheers!
Navina


On Mon, Aug 3, 2015 at 2:26 PM, Chen Song <chen.song.82@gmail.com> wrote:

> Dump question.
>
> When running samza job on YARN, I found that container logs gone after
> days. Is there a YARN config to keep the logs from being deleted for
> long-lived streaming job?
>
> --
> Chen Song
>



-- 
Navina R.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message