hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris K Wensel <ch...@wensel.net>
Subject Re: Defining log4j settings during the submit of a job
Date Wed, 09 Mar 2011 18:44:29 GMT

I'm not exactly sure how this is related to Cascading. Cascading inherits and logs via the
same APIs has Hadoop.

That said, Cascading has the ability to force a log configuration on a cluster side task,
it was non-obvious to me how I can influence them via a properties file in the classpath so
I use the log4j API directly.

The code that does that is this:

import org.apache.log4j.Level;
import org.apache.log4j.Logger;

Logger.getLogger( "package" ).setLevel( Level.toLevel( "level" ) );

you might need to do the same in your client side code (Hadoop jar main function).

ckw


On Mar 9, 2011, at 8:05 AM, Chris Curtin wrote:

> Hi,
> 
> This is a little specific to Cascading and we had a good discussion about it
> on their list, but couldn't come up with a solution.
> 
> For those who don't know, Cascading does a lot of planning and takes control
> of executing one or more map/reduce steps on the cluster. It does a lot of
> work (and allows us to write Java between map/reduce steps) on the node
> where the job is submitted, not out in the data nodes.
> 
> The issue I am having is that since the jobs are being launched via the
> hadoop jar <abc.jar> command line, the log4j settings that are being used
> are being picked up from the hadoop settings. What I want is to define the
> logging for the hadoop startup and Cascading parts to be controlled by the
> job.
> 
> For example, group A submits a job and by the default the logs go to
> console. Group B submits their own job and again it goes to console. If I
> want them to go to a log that is setup for rotation/auto purging etc. hadoop
> only allows me to do this in the hadoop log4j settings, but everything goes
> to one log file and I don't want to have to weed through all the logs to
> find information for a specific group.
> 
> To be clear, I want the logs from the machine where the  'hadoop
> jar' command was run to go to a different location, NOT the logs from when
> the job is running on cluster.
> 
> What I've tried is to put my own log4j.properties file in various locations
> in the jar, expicitly set the class path etc. None of it works since Hadoop
> forces the first item in the class path before calling the jar to be
> HADOOP_HOME.
> 
> Is there any way to define at submittal time the logging settings?
> 
> Yes I know I could modify the hadoop log4j.properties file with class-level
> appenders, but that would be a major pain as each group and new application
> comes along.
> 
> Thanks for your help,
> 
> Chris

--
Chris K Wensel
chris@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support, and licensing for Cascading


Mime
View raw message