hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Curtin <curtin.ch...@gmail.com>
Subject Defining log4j settings during the submit of a job
Date Wed, 09 Mar 2011 16:05:47 GMT

This is a little specific to Cascading and we had a good discussion about it
on their list, but couldn't come up with a solution.

For those who don't know, Cascading does a lot of planning and takes control
of executing one or more map/reduce steps on the cluster. It does a lot of
work (and allows us to write Java between map/reduce steps) on the node
where the job is submitted, not out in the data nodes.

The issue I am having is that since the jobs are being launched via the
hadoop jar <abc.jar> command line, the log4j settings that are being used
are being picked up from the hadoop settings. What I want is to define the
logging for the hadoop startup and Cascading parts to be controlled by the

For example, group A submits a job and by the default the logs go to
console. Group B submits their own job and again it goes to console. If I
want them to go to a log that is setup for rotation/auto purging etc. hadoop
only allows me to do this in the hadoop log4j settings, but everything goes
to one log file and I don't want to have to weed through all the logs to
find information for a specific group.

To be clear, I want the logs from the machine where the  'hadoop
jar' command was run to go to a different location, NOT the logs from when
the job is running on cluster.

What I've tried is to put my own log4j.properties file in various locations
in the jar, expicitly set the class path etc. None of it works since Hadoop
forces the first item in the class path before calling the jar to be

Is there any way to define at submittal time the logging settings?

Yes I know I could modify the hadoop log4j.properties file with class-level
appenders, but that would be a major pain as each group and new application
comes along.

Thanks for your help,


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message