hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maria <linanmengxia...@126.com>
Subject Re:Re:Re: Fw:Re:How to add custom field to hadoop MR task log?
Date Sat, 05 Nov 2016 06:37:15 GMT


Also,I was trying to reload $HADOOP_HOME/etc/conf/log4j.properties which I edited with <[ID:
%X{ID}] > into log4j.appender.RFA.layout.ConversionPattern in Mapper.setup(), like this:
"PropertyConfigurator.configure("/opt/software/hadoop/etc/hadoop/log4j.properties");"
It seems that $HADOOP_HOME/etc/hadoop/log4j.properties loaded,BUT log.info("*****") in Mapper.map()
can't print out in map task logs...In other words,there was no log message in map task logs
from 8088 history port..



At 2016-11-05 13:46:43, "Maria" <linanmengxiaoai@126.com> wrote:
>
>
>Hi Ravi, Thanks for your remind.
>
>Yes, as what you said, I am not sure that. I think this is a difficult problem for me.
>I really have no idea about how to confirm the log4j.properties file.
>
>Liberty of disturbance to you,can you please give me an advice about how to confirm
if the log4j.properties file is what I edited, 
>or is there any other way to add the <ID> to log message except directly  written
to LOG.info().
>
>Thank you .
>
>_Maria
>
>
>At 2016-11-05 11:35:02, "Ravi Prakash" <ravihadoop@gmail.com> wrote:
> 
>
>
>
>Hi Maria!
>
>You have to be careful which log4j.properties file is on the classpath of the task which
was launched. Often times there are multiple log4j.properties file, perhaps in the classpaths
or in one of the jars on the classpath. Are you sure the log4j.properties file you edited
is the only one loaded by the classloader?
>
>Ravi
>
>
>
>On Fri, Nov 4, 2016 at 5:06 AM, Maria <linanmengxiaoai@126.com> wrote:
>
>
>
>Sorry,(clerical errors) .
>I just modified “log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n” 
>to "log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: [ID: %X{ID}]  %m%n",
>and use "MDC.put("ID",ID)" in mapper class, It does not work.
>
>
>
>At 2016-11-04 17:01:16, "Maria" <linanmengxiaoai@126.com> wrote:
>>
>>I know that, A simple way is to write "<custom ID> " to every LOG.info()/LOG.warn()....like
this:
>>
>>logger.info(ID + " start map logic");
>>BUT,every LOG info has to add "ID" is not wise.   
>>Or else, can someone know how to modify the mapreduce task ConversionPattern configuration?
>>I tried to modify "RFA" Appender to this:
>>-------------------
>>log4j.appender.RFA=org.apache.log4j.RollingFileAppender
>>log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
>>log4j.appender.RFA.MaxFileSize=${hadoop.log.maxfilesize}
>>log4j.appender.RFA.MaxBackupIndex=${hadoop.log.maxbackupindex}
>>log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
>># Pattern format: Date LogLevel LoggerName LogMessage
>>log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
>># Debugging Pattern format
>>#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %l - [ID: %X{ID}]  %m%n
>>-----------------
>>It does not work.
>>
>>At 2016-11-04 11:26:57, "Maria" <linanmengxiaoai@126.com> wrote:
>>>
>>>Hi, dear developers,
>>>
>>>I'm trying to reconfig $HADOOP/etc.hadoop/log4j.properties, 
>>>I want to add an <custom ID> to mapreduce log before LOGmessage. Like this:
>>>"ID:234521 start map logic" 
>>>
>>>My steps as follow:
>>>(1)In my Mapper Class:
>>>
>>>static Logger logger = LoggerFactory.getLogger(Mapper.class);
>>>....
>>>
>>>public void map(Object key, Text value, Context context) throws IOException, InterruptedException
{
>>>
>>>    MDC.put("ID", "operatorID");
>>>    logger.info("start map logic");
>>>
>>>     StringTokenizer itr = new StringTokenizer(value.toString());
>>>     while (itr.hasMoreTokens()) {
>>>       word.set(itr.nextToken());
>>>       context.write(word, one);
>>>     }
>>>   }
>>> }
>>>(2)config $HADOOP/etc.hadoop/log4j.properties
>>>
>>>log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender 
>>>log4j.appender.TLA.taskId=${hadoop.tasklog.taskid} 
>>>log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup} 
>>>log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize} 
>>>   
>>>log4j.appender.TLA.layout=org.apache.log4j.PatternLayout 
>>>log4j.appender.TLA.layout.ConversionPattern=%l  %p %c: ID:[%X{ID}]  %m%n 
>>>
>>>
>>>BUT it does not work. and because use slf4j API, so I don't know how to get Appenders.
>>>
>>>I am desperately in need。。
>>>Any help would be highly appreciated
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>>For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
>>【网易自营|30天无忧退货】日本匠心设计秋冬宠物用品,限时9.9元起,还可叠加双11折扣>>
>>    
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: user-help@hadoop.apache.org
>
>
>
> 
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
Mime
View raw message