flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LINZ, Arnaud" <AL...@bouyguestelecom.fr>
Subject RE: Best way for simple logging in jobs?
Date Mon, 31 Aug 2015 15:37:45 GMT
For unknown reasons, the stdout/stderr output of my jobs wasn’t retrieved by Yarn. Same
thing for slf4j logger : outside local cluster mode, I could not see any trace from the nodes.
 I’ve spent a few hours trying to find why, but I gave up.
Since I need “real time” logging & monitoring in the main driver application, I’ve
ended up implementing a simple push/pull socket message queue using JeroMQ (pure-Java 0MQ).
That allows me to easily publish log & metrology information from the worker nodes to
the driver program.

De : ewenstephan@gmail.com [mailto:ewenstephan@gmail.com] De la part de Stephan Ewen
Envoyé : lundi 31 août 2015 11:05
À : user@flink.apache.org
Objet : Re: Best way for simple logging in jobs?


Are you looking for a separate user log file next to the system log file, or would Robert's
suggestion work?

On Fri, Aug 28, 2015 at 4:20 PM, Robert Metzger <rmetzger@apache.org<mailto:rmetzger@apache.org>>

Creating a slf4j logger like this:

private static final Logger LOG = LoggerFactory.getLogger(PimpedKafkaSink.class);

Works for me. The messages also end up in the regular YARN logs.

Also system out should end up in YARN actually (when retrieving the logs from the log aggregation).



On Fri, Aug 28, 2015 at 3:55 PM, LINZ, Arnaud <ALINZ@bouyguestelecom.fr<mailto:ALINZ@bouyguestelecom.fr>>


I am wondering if it’s possible to get my own logs inside the job functions (sources, mappers,
sinks…).  It would be nice if I could get those logs in the Yarn’s logs, but writing System.out/System.err
has no effect.

For now I’m using a “StringBuffer” accumulator but it does not work in streaming apps
before v0.10, and only show results at the end.

I’ll probably end up using a HDFS logging system but there is maybe a smarter way ?




L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice
ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation
ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message,
merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent
this message cannot therefore be held liable for its content nor attachments. Any unauthorized
use or dissemination is prohibited. If you are not the intended recipient of this message,
then please delete it and notify the sender.

View raw message