apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukkamula, Suryavamshivardhan (CWM-NR)" <suryavamshivardhan.mukkam...@rbc.com>
Subject RE: problems viewing the Datatorrent application logs
Date Thu, 23 Mar 2017 15:47:34 GMT
Thank you very much Ram.

Regards,
Surya Vamshi

From: Munagala Ramanath [mailto:ram@datatorrent.com]
Sent: 2017, March, 23 10:31 AM
To: users@apex.apache.org
Subject: Re: problems viewing the Datatorrent application logs

If log file aggregation is enabled, aggregated logs for terminated logs should be available
on HDFS and retrievable with:
    yarn logs -applicationId <application ID>

Log aggregation is discussed further at: https://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
and is controlled by the following configuration parameters:

yarn.log-aggregation-enable
yarn.log-aggregation.retain-check-interval-seconds
yarn.log-aggregation.retain-seconds
yarn.nodemanager.log-aggregation.compression-type
yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds

For running applications, logs are not on HDFS but rather on the local filesystems of the
individual nodes; the location
depends on the following configuration parameters:

yarn.nodemanager.remote-app-log-dir
yarn.log.dir
yarn.nodemanager.log-dirs

Some example locations are discussed in the Beginner's Guide at: http://docs.datatorrent.com/beginner/
in the section entitled: Checking logs

Ram

On Thu, Mar 23, 2017 at 6:43 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <suryavamshivardhan.mukkamula@rbc.com<mailto:suryavamshivardhan.mukkamula@rbc.com>>
wrote:
Hello Team,

One of our data torrent application , we cannot view the logs from web console due to firewall
issues(Datatorrent installed by service user ‘A’ and application is being launched by
service user ‘B’).

Can you please let us know where can I find the path of the application logs (running and
history jobs) either on the HDFS or individual nodes ? , we have yarn logs from the Hadoop
web console but due to the same fire wall issues we can only view the logs through  window
server of same domain and it is very painful and time consuming.

While the platform team trying to fix the issue , can you please let us know where can I find
the log files on the HDFS or individual nodes.

Regards,
Surya Vamshi




_______________________________________________________________________

If you received this email in error, please advise the sender (by return email or otherwise)
immediately. You have consented to receive the attached electronically at the above-noted
email address; please retain a copy of this confirmation for future reference.

Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par
retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s)
ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver
une copie de cette confirmation pour les fins de reference future.



--


_______________________________________________________

Munagala V. Ramanath

Software Engineer

E: ram@datatorrent.com<mailto:ram@datatorrent.com> | M: (408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com<http://www.datatorrent.com>  |  apex.apache.org<http://apex.apache.org>

[https://lh4.googleusercontent.com/gZ-_bA3cXH8UerggUOqWVpDglJsOiKtL3dpoCCWrX0K1YEYQHbJUoR106D3gztJVPaRyfeuQVMLKM3PCDfZo-wzBaliedZdlaL5YSqmZb_Weje_De0qIPsLSMAgzRZiE5lIECrT4]

_______________________________________________________________________
If you received this email in error, please advise the sender (by return email or otherwise)
immediately. You have consented to receive the attached electronically at the above-noted
email address; please retain a copy of this confirmation for future reference.  

Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par
retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s)
ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver
une copie de cette confirmation pour les fins de reference future.
Mime
View raw message