hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7815) Mount the filecache as read-only in Docker containers
Date Mon, 29 Jan 2018 20:55:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344007#comment-16344007

Jason Lowe commented on YARN-7815:

bq. Would it make sense to detach the appcache and mount a separate appcache dir for each
container? AFAIK it is not for sharing between containers, since they might get scheduled
to other nodes anyways.

It is used for sharing in some circumstances, e.g.: Tez shared fetch where a task can avoid
fetching a broadcast output that another task already fetched, or Tez local fetch where a
downstream task that runs on the same node fetches an output directly from local disk rather
than having it copied through the shuffle server.  Besides those existing use-cases, having
a separate appcache directory per container would add significant load to the shuffle handler,
since it would add another dimension to the search matrix for shuffle data.

Bottom line is we have to mount the application's appcache directory read/write for backwards
compatibility.  I don't see that as being a big concern, as compromising a single container
is already compromising the entire application (due to the application secrets available within
that container).  The key is preventing access/corruption to other applications even from
the same user.

I think that leaves us with this proposal which should accomplish that and remove one of the
mounts being made today:

1. nm-local-dir/filecache mounted read-only for access to localized public files
2. nm-local-dir/usercache/_user_/filecache mounted read-only for access to localized user-private
3. nm-local-dir/usercache/_user_/appcache/_applicationId_ mounted read-write for access to
the application work area and underlying container working directory

> Mount the filecache as read-only in Docker containers
> -----------------------------------------------------
>                 Key: YARN-7815
>                 URL: https://issues.apache.org/jira/browse/YARN-7815
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Shane Kumpf
>            Assignee: Shane Kumpf
>            Priority: Major
> Currently, when using the Docker runtime, the filecache directories are mounted read-write
into the Docker containers. Read write access is not necessary. We should make this more restrictive
by changing that mount to read-only.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message