gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karol Brejna (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (GEARPUMP-161) Application logs are not created on Yarn
Date Mon, 12 Dec 2016 12:58:58 GMT

     [ https://issues.apache.org/jira/browse/GEARPUMP-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Karol Brejna resolved GEARPUMP-161.
-----------------------------------
       Resolution: Fixed
         Assignee: Karol Brejna  (was: Huafeng Wang)
    Fix Version/s: 0.8.0

The problem was the way we invoked yarnclient.

I've just tested running it in docker and setting the user to one that is well configured
on hadoop side (it was not the case before) and the logfile is now properly merged (log4j
properties created for yarn and default log4j properties).

```
krb-aws 12:00:16 root@hadoop-worker-2.instance.cluster.local:/data/yl/usercache/broker/appcache/application_1481493278525_0003/container_e02_1481493278525_0003_01_000003/conf
]# cat log4j.properties
#gearpump on yarn
#Mon Dec 12 08:34:19 UTC 2016
log4j.appender.ApplicationLogAppender.layout.ConversionPattern=[%p] [%d{MM/dd/yyyy HH\:mm\:ss.SSS}]
[%c{1}] %m%n
log4j.appender.ApplicationLogAppender.MaxFileSize=100MB
log4j.appender.ApplicationLogAppender.File=${gearpump.application.log.dir}/${gearpump.application.log.file}
log4j.appender.RollingFileAppender.layout.ConversionPattern=%d{ISO8601} %p %c{1}\: %m%n
gearpump.application.log.file=executor${gearpump.executorId}-${JVM_NAME}.log
log4j.appender.ApplicationLogAppender.layout=org.apache.log4j.PatternLayout
gearpump.log.file=gearpump.log
log4j.appender.RollingFileAppender.checkFileInterval=60
log4j.appender.RollingFileAppender.File=${gearpump.log.dir}/${gearpump.log.file}
gearpump.username=user
gearpump.root.logger=RollingFileAppender
gearpump.master.log.file=gearpump-master-${JVM_NAME}.log
gearpump.ui.log.file=gearpump-ui-${JVM_NAME}.log
log4j.rootLevel=INFO
gearpump.application.log.rootdir=logs
gearpump.application.log.dir=${gearpump.application.log.rootdir}/applicationData/${gearpump.username}/${gearpump.master.starttime}/app${gearpump.applicationId}
log4j.appender.console.layout.ConversionPattern=[%c{1}] %m%n
log4j.rootAppender=console
log4j.appender.RollingFileAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.ApplicationLogAppender.checkFileInterval=60
log4j.appender.console.layout=org.apache.log4j.PatternLayout
gearpump.application.logger=ApplicationLogAppender
gearpump.local.log.file=gearpump-local-${JVM_NAME}.log
gearpump.log.dir=logs
log4j.appender.RollingFileAppender.MaxBackupIndex=10
log4j.appender.console=org.apache.log4j.ConsoleAppender
gearpump.worker.log.file=gearpump-worker-${JVM_NAME}.log
log4j.threshhold=ALL
log4j.appender.ApplicationLogAppender.MaxBackupIndex=30
log4j.appender.RollingFileAppender=io.gearpump.util.RecreateRollingFileAppender
log4j.rootLogger=${log4j.rootLevel},${log4j.rootAppender}
log4j.appender.RollingFileAppender.MaxFileSize=200MB
log4j.appender.ApplicationLogAppender=io.gearpump.util.RecreateRollingFileAppender```

> Application logs are not created on Yarn 
> -----------------------------------------
>
>                 Key: GEARPUMP-161
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-161
>             Project: Apache Gearpump
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 0.8.0
>         Environment: yarn deployment: 1 master, 1 worker;
>            Reporter: Karol Brejna
>            Assignee: Karol Brejna
>             Fix For: 0.8.0
>
>
> I have a simple cluster (1 master, 1 worker) running on YARN.
> I deploy an app. The app lands on a worker (cdh-worker-2 machine in this case).
> I inspect the worker and the app settings in the dashborad.
> Worker dir is shown as: /var/log/hadoop-yarn/container/application_1466296175601_0020/container_e05_1466296175601_0020_01_000003
> App log dir: /data1/hadoop/yarn/nm/usercache/cf/appcache/application_1466296175601_0020/container_e05_1466296175601_0020_01_000003/logs
> The logs are written to worker log dir under stderr.
> But there is no such a folder for app logs. 'logs' doesn exists under indicated location.
> I tried to do a wrokarounds (as I thought it may be a simple permissions issue):
> 1.
> chmod 777 on parent dir (/data1/hadoop/yarn/nm/usercache/cf/appcache/application_1466296175601_0020/container_e05_1466296175601_0020_01_000003)
> restarted app 
> No results here. logs dir not created.
> 2. 
> created logs dir by hand and changed permissions:
> mkdir logs
> chmod 777 logs
> No results here. logs dir not populated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message