flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fritz Budiyanto <fbudi...@icloud.com>
Subject Re: Excessive stdout is causing java heap out of mem
Date Tue, 23 May 2017 05:02:45 GMT
Hi Robert,

Thanks Robert, I’ll start using the logger. 

I didn’t pay attention whether the error occur when I accessed the log from job manager.
I will do that in my next test.

Anyone has any suggestion on how to debug out of memory exception on flink jm/tm ?

—
Fritz


> On May 22, 2017, at 12:04 PM, Robert Metzger <rmetzger@apache.org> wrote:
> 
> Hi Fritz,
> 
> The TaskManagers are not buffering all stdout for the webinterface (at least I'm not
aware of that). Did the error occur when accessing the log from the JobManager?
> Flinks web front end lazily loads the logs from the taskmanagers.
> 
> The suggested method for logging is to use slf4j for logging, so the following code snippets
:
> 
> import org.slf4j.Logger;
> import org.slf4j.LoggerFactory;
> private static final Logger LOG = LoggerFactory.getLogger(MyJob.class);
> Then you can do stuff like:
> LOG.info("My log statement");
> Also, using a logging Framework will allow you to redirect the log contents of your job
to a separate file.
> 
> But I'm not sure if the logging is really causing the TaskManager JVMs to die ...
> 
> 
> On Sat, May 20, 2017 at 3:12 AM, Fritz Budiyanto <fbudiyan@icloud.com <mailto:fbudiyan@icloud.com>>
wrote:
> Hi,
> 
> I notice that when I enabled DataStreamSink’s print() for debugging, (kinda excessive
printing), its causing java Heap out of memory.
> Possibly the Task Manager is buffering all stdout for the WebInterface? I haven’t spent
time debugging it, but I wonder if this is expected where massive print will exhaust java
heap, and I’m using standalone mode.
> 
> Is there a way to disable this memory logging for web interface, and just redirect stdout
to file instead with file rotation?
> What is the suggested method of logging ?
> 
> —
> Fritz
> 


Mime
View raw message