hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Badger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
Date Tue, 01 Aug 2017 14:19:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108978#comment-16108978

Eric Badger commented on YARN-6494:

bq. At a minimum, we'd need the ability to turn this off for containers that don't need the
HDFS socket, but I feel that it would be better to have a more holistic approach, which is
what I hope YARN-5534 can become. The reason I believe we shouldn't hard code mounts is that
not every container will require that mount.
I generally agree with this approach, but adding the volume whitelist doesn't fix the potential
security issue. It only sort of mitigates it. If the administrator allows for the socket to
be in the whitelist, then any container can ask for it. So yes, existing containers and/or
containers that the attacker does not control upon startup will not have this bind-mounted
in. However, if the attacker is the one submitting the job, they'll just ask for the socket
to be bind-mounted and will be granted that request. Basically what I'm trying to get at is
that if the administrator allows short-circuit reads, they are taking the potential security
risk. At that point, I'm not sure if it matters whether all containers have the socket or
just the ones that asked for it, especially when the attacker can explicitly ask for it. 

I think both points can be resolved by letting the administrator decide their destiny here.
We can use YARN-5534 to create a whitelist of volumes that the jobs can specify. Then, as
we touched upon in [this comment|https://issues.apache.org/jira/browse/YARN-5534?focusedCommentId=16093026&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16093026]
in YARN-5534, we can create a default bind list, which is empty by default. If the administrator
wants the short-circuit socket for all containers, they can add it to the default list. If
they only want it for certain containers, they can add it to the whitelist and let users ask
for it. 

> add mounting of HDFS Short-Circuit path for docker containers
> -------------------------------------------------------------
>                 Key: YARN-6494
>                 URL: https://issues.apache.org/jira/browse/YARN-6494
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Jaeboo Jeong
>            Assignee: Jaeboo Jeong
>         Attachments: YARN-6494.001.patch, YARN-6494.002.patch
> Currently there is a error message about HDFS short-circuit when docker container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when trying to
connect to ‘xxxxxxx’
>     at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
>     at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
>     at org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
>     at org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't equal “”,
we need to mount volume for short-circuit path.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message