flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8247) Support Hadoop-free variant of Flink on Mesos
Date Fri, 14 Sep 2018 12:38:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614779#comment-16614779
] 

ASF GitHub Bot commented on FLINK-8247:
---------------------------------------

yanghua commented on a change in pull request #6689: [FLINK-8247] [Mesos] Support case where
hadoop is not on classpath
URL: https://github.com/apache/flink/pull/6689#discussion_r217697261
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/HadoopUserOverlay.java
 ##########
 @@ -57,24 +57,58 @@ public void configure(ContainerSpecification container) throws IOException
{
 	}
 
 	public static Builder newBuilder() {
-		return new Builder();
+		// First check if we have Hadoop in the ClassPath. If not, we simply don't do anything.
+		try {
+			Class.forName(
+				"org.apache.hadoop.security.UserGroupInformation",
+				false,
+				HadoopUserOverlay.class.getClassLoader());
+		} catch (ClassNotFoundException e) {
+			LOG.info("Cannot create Hadoop User Overlay because Hadoop cannot be found in the Classpath.");
+			return new NoOpHadoopUserOverlayBuilder();
+		}
+
+		return new HadoopUserOverlayBuilder();
 	}
 
 	/**
 	 * A builder for the {@link HadoopUserOverlay}.
 	 */
-	public static class Builder {
+	public interface Builder {
+		/**
+		 * Configures the overlay using the current Hadoop user information (from {@link UserGroupInformation}).
+		 */
+		Builder fromEnvironment(Configuration globalConfiguration) throws IOException;
+
+		HadoopUserOverlay build();
+	}
+
+	/**
+	 * A builder for {@link HadoopUserOverlay} for when hadoop doesn't exist on the classpath
+	 */
+	public static class NoOpHadoopUserOverlayBuilder implements Builder {
 
 Review comment:
   what's the meaning of "Op" in class name `NoOpHadoopUserOverlayBuilder `, maybe `NoHadoopUserOverlayBuilder`
or `EmptyHadoopUserOverlayBuilder` looks better, just personal opinion.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Support Hadoop-free variant of Flink on Mesos
> ---------------------------------------------
>
>                 Key: FLINK-8247
>                 URL: https://issues.apache.org/jira/browse/FLINK-8247
>             Project: Flink
>          Issue Type: Bug
>          Components: Mesos
>    Affects Versions: 1.4.0
>            Reporter: Eron Wright 
>            Assignee: Eron Wright 
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.4.3, 1.5.4
>
>
> In Hadoop-free mode, Hadoop isn't on the classpath.  The Mesos job manager normally uses
the Hadoop UserGroupInformation class to overlay a user context (`HADOOP_USER_NAME`) for the
task managers.  
> Detect the absence of Hadoop and skip over the `HadoopUserOverlay`, similar to the logic
in `HadoopModuleFactory`.    This may require the introduction of an overlay factory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message