Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
From: StephanEwen <git@git.apache.org>
To: issues@flink.incubator.apache.org
Reply-To: issues@flink.incubator.apache.org
References: <git-pr-1524-flink@git.apache.org>
In-Reply-To: <git-pr-1524-flink@git.apache.org>
Subject: [GitHub] flink pull request: FLINK-2380: allow to specify the default
 files...
Content-Type: text/plain
Message-Id: <20160128135015.6FB6BDFF8E@git1-us-west.apache.org>
Date: Thu, 28 Jan 2016 13:50:15 +0000 (UTC)

Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1524#discussion_r51123709
  
    --- Diff: flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java ---
    @@ -159,56 +163,109 @@ public int hashCode() {
     	/**
     	 * Returns a reference to the {@link FileSystem} instance for accessing the
     	 * local file system.
    -	 * 
    +	 *
     	 * @return a reference to the {@link FileSystem} instance for accessing the
    -	 *         local file system.
    +	 * local file system.
     	 */
     	public static FileSystem getLocalFileSystem() {
     		// this should really never fail.
     		try {
     			URI localUri = OperatingSystem.isWindows() ? new URI("file:/") : new URI("file:///");
     			return get(localUri);
    -		}
    -		catch (Exception e) {
    +		} catch (Exception e) {
     			throw new RuntimeException("Cannot create URI for local file system");
     		}
     	}
     
     	/**
    +	 * The default filesystem scheme to be used. This can be specified by the parameter
    +	 * <code>fs.default-scheme</code> in <code>flink-conf.yaml</code>. By default this is
    +	 * set to <code>file:///</code> (see {@link ConfigConstants#FILESYSTEM_SCHEME}
    +	 * and {@link ConfigConstants#DEFAULT_FILESYSTEM_SCHEME}), and uses the local filesystem.
    +	 */
    +	private static URI defaultScheme;
    +
    +	/**
    +	 * <p>Sets the default filesystem scheme based on the user-specified configuration parameter
    +	 * <code>fs.default-scheme</code>. By default this is set to <code>file:///</code>
    +	 * (see {@link ConfigConstants#FILESYSTEM_SCHEME} and
    +	 * {@link ConfigConstants#DEFAULT_FILESYSTEM_SCHEME}),
    +	 * and the local filesystem is used.
    +	 * <p>
    +	 * As an example, if set to <code>hdfs://localhost:9000/</code>, then an HDFS deployment
    +	 * with the namenode being on the local node and listening to port 9000 is going to be used.
    +	 * In this case, a file path specified as <code>/user/USERNAME/in.txt</code>
    +	 * is going to be transformed into <code>hdfs://localhost:9000/user/USERNAME/in.txt</code>. By
    +	 * default this is set to <code>file:///</code> which points to the local filesystem.
    +	 * @param config the configuration from where to fetch the parameter.
    +	 */
    +	public static void setDefaultScheme(Configuration config) throws IOException {
    +		synchronized (SYNCHRONIZATION_OBJECT) {
    +			if (defaultScheme == null) {
    +				String stringifiedUri = config.getString(ConfigConstants.FILESYSTEM_SCHEME,
    +					ConfigConstants.DEFAULT_FILESYSTEM_SCHEME);
    +				try {
    +					defaultScheme = new URI(stringifiedUri);
    +				} catch (URISyntaxException e) {
    +					throw new IOException("The URI used to set the default filesystem " +
    +						"scheme ('" + stringifiedUri + "') is not valid.");
    +				}
    +			}
    +		}
    +	}
    +
    +	/**
    +	 * <p><b>ATTENTION:</b> this method is only used in tests.</p>
    +	 * <p>Clears the previously set <code>fs.default-scheme</code>
    +	 * (see {@link ConfigConstants#FILESYSTEM_SCHEME})</p>
    +	 */
    +	public static void clearDefaultScheme() {
    --- End diff --
    
    This should not be public. Users do not read comments and simply call this. The tests probably need a custom reflection cleaner to reset that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---