apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2077) SingleFileOutputOperator should append partitionId to file name
Date Mon, 16 May 2016 23:34:12 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285653#comment-15285653
] 

ASF GitHub Bot commented on APEXMALHAR-2077:
--------------------------------------------

Github user amberarrow commented on a diff in the pull request:

    https://github.com/apache/incubator-apex-malhar/pull/273#discussion_r63443902
  
    --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/AbstractSingleFileOutputOperator.java
---
    @@ -39,10 +41,37 @@
       @NotNull
       protected String outputFileName;
     
    +  /**
    +   * partitionedFileName string format specifier 
    +      e.g. fileName_physicalPartionId -> %s_%d 
    +   */
    +  private String partitionedFileNameformat = "%s_%d";
    +
    +  /**
    +   * Derived name for file based on physicalPartitionId
    +   */
    +  private transient String partitionedFileName;
    +
    +  /**
    +   * Physical partition id for the current partition.
    +   */
    +  private transient int physicalPartitionId;
    +
    +  /**
    +   * Initializing current partition id, partitionedFileName etc. {@inheritDoc}
    +   */
    +  @Override
    +  public void setup(OperatorContext context)
    --- End diff --
    
    Yogi, if you can just implement the check if the format string is null or empty, we can
merge this.  We can defer the issue of whether dynamically changing the file name is supported
for the next release. We have no guarantee on the value of the operator id (not guaranteed
to start at 0).


> SingleFileOutputOperator should append partitionId to file name
> ---------------------------------------------------------------
>
>                 Key: APEXMALHAR-2077
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2077
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>            Reporter: Yogi Devendra
>            Assignee: Yogi Devendra
>            Priority: Minor
>             Fix For: 3.4.0
>
>
> If SingleFileOutputOperator is partitioned into more than one physical instances then
it  should append partitionId to file name. So that, each partition can independently write
to separate file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message