beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stas Levin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-1168) FileBaseSink: Handle output to locations such as ".", "~", or maybe ".."
Date Thu, 29 Dec 2016 14:36:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15785346#comment-15785346
] 

Stas Levin edited comment on BEAM-1168 at 12/29/16 2:36 PM:
------------------------------------------------------------

[~peihe0@gmail.com], is this issue still active?
I have tried to reproduce it using {{MinimalWordCount}} by setting the output to "." like
so: {{p.apply(TextIO.Write.to("."))}} and the output seems to be written to {{temp-beam-.-2016-12-364_13-25-50}},
with subdirectories named as UIDs (e.g. {{77715ed2-4124-4506-ae49-6a9ab447a469}}).

The output dir's name seems to be formatted like so in {{FileBasedSink.FileBasedWriteOperation.TemporaryDirectoryBuilder#apply(...)}}:
{code:java}
baseOutputPath
              .resolveSibling(
                  "temp-beam-"
                  + baseOutputPath.getFileName()
                  + "-"
                  + now.toString(DateTimeFormat.forPattern("yyyy-MM-DD_HH-mm-ss")))

{code}

If {{".","..","~"}} are provided they will be embedded inside the {{temp-beam....}} string
without creating hidden files. The leaf files inside this directory are named after UIDs so
they will not be hidden as well.  Am I missing something?


was (Author: staslev):
[~peihe0@gmail.com], is this issue till active?
I have tried to reproduce it using {{MinimalWordCount}} by setting the output to "." like
so: {{p.apply(TextIO.Write.to("."))}} and the output seems to be written to {{temp-beam-.-2016-12-364_13-25-50}},
with subdirectories named as UIDs (e.g. {{77715ed2-4124-4506-ae49-6a9ab447a469}}).

The output dir's name seems to be formatted like so in {{FileBasedSink.FileBasedWriteOperation.TemporaryDirectoryBuilder#apply(...)}}:
{code:java}
baseOutputPath
              .resolveSibling(
                  "temp-beam-"
                  + baseOutputPath.getFileName()
                  + "-"
                  + now.toString(DateTimeFormat.forPattern("yyyy-MM-DD_HH-mm-ss")))

{code}

If {{".","..","~"}} are provided they will be embedded inside the {{temp-beam....}} string
without creating hidden files. The leaf files inside this directory are named after UIDs so
they will not be hidden as well.  Am I missing something?

> FileBaseSink: Handle output to locations such as ".", "~", or maybe ".."
> ------------------------------------------------------------------------
>
>                 Key: BEAM-1168
>                 URL: https://issues.apache.org/jira/browse/BEAM-1168
>             Project: Beam
>          Issue Type: Sub-task
>          Components: sdk-java-core
>            Reporter: Pei He
>            Priority: Minor
>
> Currently, WordCount will write to hidden files if output is set to "." or "..".
> .-00002-of-00005
> ..-00002-of-00005
> Or ~-00002-of-00005, if the output is set to "~".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message