hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-15722) regression: Hadoop 2.7.7 release breaks spark submit
Date Mon, 24 Sep 2018 09:44:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16625579#comment-16625579
] 

Brahma Reddy Battula edited comment on HADOOP-15722 at 9/24/18 9:43 AM:
------------------------------------------------------------------------

{quote} Looks like in the following log snippets for the good and bad runs, the user.name
system property is not getting expanded in the bad run because the conf resource is untrusted:
{quote}
Yes,Our Spark Team also faced the same issue and analysed the issue. Please check following
for same.

if *hive hive.exec.scratchdir* is set to */tmp/hive-${user.name}* when we can see the effect
 hive dosAs is default true, so spark will execute hive as is if its proxy user
 First export proxy user to user_a
 when *user_a* creates a scratch dir, he will create {color:#205081}*/tmp/hive-*{color}{color:#d04437}*$%7Buser.name%7D*{color}
(failure to replace system property in path in 2.7.7) instead of /tmp/hive-user_a (replace
system property in path in 2.7.3)
 Second export proxy user to user_b
 when *user_b* tries to create scratch dir,{color:#ff0000} {color:#333333}he will fail as
he will also try to create in{color}{color}{color:#205081} */tmp/hive*-{color}{color:#d04437}*$%7Buser.name%7D*{color} instead
of {color:#14892c}*/tmp/hive-user_b* {color}

 

IMO,Usually we'll configure system properties in xml's. So I feel,we should allow system properties(option2
should be fine?).

Verified with "allowing system properties", it was working fine.

This should be blocker for 3.2 release..?  If you all agree, we can mark this as blocker.


was (Author: brahmareddy):
{quote} Looks like in the following log snippets for the good and bad runs, the user.name
system property is not getting expanded in the bad run because the conf resource is untrusted:
{quote}
Yes,Our Spark Team also faced the same issue and analysed the issue. Please check following
for same.

if *hive hive.exec.scratchdir* is set to */tmp/hive-${user.name}* when we can see the effect
 hive dosAs is default true, so spark will execute hive as is if its proxy user
 First export proxy user to user_a
 when *user_a* creates a scratch dir, he will create {color:#205081}*/tmp/hive-*{color}{color:#d04437}*$%7Buser.name%7D*{color}
(failure to replace system property in path in 2.7.7) instead of /tmp/hive-user_a (replace
system property in path in 2.7.3)
 Second export proxy user to user_b
 when *user_b* tries to create scratch dir,{color:#ff0000} {color:#333333}he will fail as
he will also try to create in{color}{color} {color:#205081}*/tmp/hive*{color}-*$%7Buser.name%7D* instead
of {color:#14892c}*/tmp/hive-user_b* {color}

 

IMO,Usually we'll configure system properties in xml's. So I feel,we should allow system properties(option2
should be fine?).

Verified with "allowing system properties", it was working fine.

This should be blocker for 3.2 release..?  If you all agree, we can mark this as blocker.

> regression: Hadoop 2.7.7 release breaks spark submit
> ----------------------------------------------------
>
>                 Key: HADOOP-15722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, conf, security
>    Affects Versions: 2.7.7
>            Reporter: Steve Loughran
>            Priority: Major
>
> SPARK-25330 highlights that upgrading spark to hadoop 2.7.7 is causing a regression in
client setup, with things only working when {{Configuration.getRestrictParserDefault(Object
resource)}} = false.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message