hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Gäde <s116...@hft-leipzig.de>
Subject Re: where to put log4j.properties for logging out of MR job?
Date Wed, 21 May 2014 17:31:45 GMT
Ok, my fault. It is not about the PatternLayout. But deleting the file Flume is currently using
without restarting the agent is not a good idea. ;-)

Cheers
Seb.

Am 19.05.2014 um 20:46 schrieb Sebastian Gäde <s116102@hft-leipzig.de>:

> Hi,
> 
> thanks, worked for me, (using http://stackoverflow.com/questions/6072389/how-to-create-a-own-appender-in-log4j):
> 
> public class MyFlumeAppender extends AppenderSkeleton {	
> MyRpcClientFacade client = new MyRpcClientFacade();
> 
> public MyFlumeAppender() {
> 	super();
> 	String flumeServer = "localhost";
> 	Configuration conf = new Configuration();
> 	conf.addResource("app-config.xml");
> 	flumeServer = conf.get("flume.server");
> 	client.init(flumeServer, 41415);
> }
> 
> @Override
> public void close() {
> 	client.cleanUp();
> }
> 
> @Override
> public boolean requiresLayout() {
> 	// TODO Auto-generated method stub
> 	return false;
> }
> 
> @Override
> protected void append(LoggingEvent event) {
> 	//String logger = event.getLoggerName();
> 	//String level = event.getLevel().toString();
> 	//Long ts = event.getTimeStamp();
> 	//client.sendDataToFlume(ts.toString()+" - "+level+" - "+event.getMessage().toString());
> 	PatternLayout myLayout = new PatternLayout("%d{ISO8601} %-5p %c - %m");
> 	client.sendDataToFlume(myLayout.format(event));
> 	}
> 
> }
> 
> However, when I try to use the PatternLayout for formatting the output, logging falls
back to the container logs. If I build a string on my own (uncommenting the first four lines
in append) it's working. I know this might be a bit off topic now, but maybe someone has an
idea?
> 
> Cheers
> Seb.
> 
> Am 19.05.2014 11:17, schrieb Harsh J:
>> Task JVMs in MR run with their own TaskLogAppender-configured
>> task-log4j.properties file. This is not overridable or configurable
>> currently.
>> 
>> Your best approach at the moment would be to configure your custom
>> appender in code at the beginning of your program.
>> 
>> On Sun, May 18, 2014 at 9:27 PM, Sebastian Gäde <s116102@hft-leipzig.de> wrote:
>>> Hi,
>>> 
>>> I'd like to log events within my MR application using Flume writing to a
>>> central log file. I've followed
>>> http://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#log4j-appender
>>> putting the stuff into Hadoop's log4j.properties and copying it to Hadoop's
>>> conf directory on all nodes.
>>> 
>>> However, my in-application events are only logged to stdout (I can see them
>>> using the YARN webapp) but not to Flume. Although I thought Hadoop's conf
>>> directory is added to classpath, it does not seem to work for me.
>>> 
>>> Could you please tell me where I should put the log4j.properties?
>>> 
>>> Cheers
>>> Seb.
>> 
>> 
>> 


Mime
View raw message