Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Thu, 6 Apr 2017 17:22:41 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13060882.1491039236000.226713.1491499361778@Atlassian.JIRA>
In-Reply-To: <JIRA.13060882.1491039236000@Atlassian.JIRA>
References: <JIRA.13060882.1491039236000@Atlassian.JIRA> <JIRA.13060882.1491039236956@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-1856) HDFSFileSink class do not use the
 same configuration in master and slave
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 06 Apr 2017 17:22:45 -0000


    [ https://issues.apache.org/jira/browse/BEAM-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959361#comment-15959361 ] 

ASF GitHub Bot commented on BEAM-1856:
--------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/beam/pull/2399


> HDFSFileSink class do not use the same configuration in master and slave
> ------------------------------------------------------------------------
>
>                 Key: BEAM-1856
>                 URL: https://issues.apache.org/jira/browse/BEAM-1856
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>    Affects Versions: 0.6.0
>            Reporter: yangping wu
>            Assignee: yangping wu
>             Fix For: First stable release
>
>
> I have a code snippet as follow:
> {code}
> Read.Bounded<KV<LongWritable, Text>> from = Read.from(HDFSFileSource.from(options.getInputFile(), TextInputFormat.class, LongWritable.class, Text.class));
> PCollection<KV<LongWritable, Text>> data = p.apply(from);
> data.apply(MapElements.via(new SimpleFunction<KV<LongWritable, Text>, String>() {
>     @Override
>     public String apply(KV<LongWritable, Text> input) {
>         return input.getValue() + "\t" + input.getValue();
>     }
> })).apply(Write.to(HDFSFileSink.<String>toText(options.getOutputFile())));
> {code}
> and submit job like this:
> {code}
> spark-submit --class org.apache.beam.examples.WordCountHDFS --master yarn-client   \
>              ./target/word-count-beam-bundled-0.1.jar                              \
>              --runner=SparkRunner                                                  \
>              --inputFile=hdfs://master/tmp/input/                                  \
>              --outputFile=/tmp/output/
> {code}
> Then {{HDFSFileSink.validate}} function will check whether the local filesystem (not HDFS) exists {{/tmp/output/}} directory.
> But the final result will store in {{hdfs://master/tmp/output/}} directory in HDFS filesystem.
> The reason is {{HDFSFileSink}} class do not use the same configuration in master thread and slave thread.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)