hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shay Rojansky <r...@roji.org>
Subject Log aggregation ownership issue with non-HDFS setup
Date Tue, 19 May 2015 15:37:27 GMT
Hi.

I'm trying to set up yarn log aggregation on a cluster using a shared NFS
filesystem (no HDFS). The issue is that the user directories that get
created under yarn.nodemanager.remote-app-log-dir are owned by the node
manager owner, and not by the submitting user (which therefore can't access
their logs). I've also tried running the node manager as root, to no avail.

Looking into the sources, it seems that the LogWriter freates the file
within a UserGroupInformation.doAs block; I'm guessing that under HDFS that
means "impersonating" the submitting user, and the file gets created with
their ownership. However, if yarn.nodemanager.remote-app-log-dir happens to
be on a file:/// filesystem and not HDFS, this doesn't happen.

Can anyone confirm this, and/or suggest a workaround? Is it maybe possible
to access the aggregated logs via some sort of rest API?

Thanks,

Shay

Mime
View raw message