samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject Re: Keep samza container logs from being deleted
Date Tue, 25 Aug 2015 21:39:36 GMT
Thanks Navina.

What I found is that containers logs were gone after days locally even when
the job is still running.

Chen

On Mon, Aug 3, 2015 at 6:06 PM, Navina Ramesh <nramesh@linkedin.com.invalid>
wrote:

> Hi Chen,
> You can set the yarn.nodemanage.delete.debug-delay-sec config in
> yarn-site.xml, in order to retain the logs after the job crashes. Refer
>
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>
>
> (This was intended for debug purpose).
>
> If you need to retain log for longer period of time for purposes other than
> debugging, you should look into yarn's log aggregation features.
> Alternatively, you can use Samza's StreamAppender to produce the log data
> to a stream. You can then publish it to ELK for further lookup. It really
> depends on your requirements.
> Please note that the StreamAppender has a bug in the master branch that is
> pending resolution in https://issues.apache.org/jira/browse/SAMZA-723
>
> Cheers!
> Navina
>
>
> On Mon, Aug 3, 2015 at 2:26 PM, Chen Song <chen.song.82@gmail.com> wrote:
>
> > Dump question.
> >
> > When running samza job on YARN, I found that container logs gone after
> > days. Is there a YARN config to keep the logs from being deleted for
> > long-lived streaming job?
> >
> > --
> > Chen Song
> >
>
>
>
> --
> Navina R.
>



-- 
Chen Song

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message