camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-6867) camel-hdfs - HdfsProducer filename collisions when Producer instance recreated
Date Sat, 19 Oct 2013 07:10:42 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799819#comment-13799819
] 

Claus Ibsen commented on CAMEL-6867:
------------------------------------

2.10 is no longer supported, so 2.11 onwards is fine.

> camel-hdfs - HdfsProducer filename collisions when Producer instance recreated
> ------------------------------------------------------------------------------
>
>                 Key: CAMEL-6867
>                 URL: https://issues.apache.org/jira/browse/CAMEL-6867
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-hdfs
>            Reporter: Ben O'Day
>            Assignee: Ben O'Day
>             Fix For: 2.13.0
>
>
> The HdfsProducer uses an instance variable (long splitNum) that is incremented to create
unique output filenames in a given directory (seg0, seg1, etc).  
> If the Producer instance is recreated (producer cache limit exceeded, server restart,
etc), the splitNum variable is reset to 0.  This results in files being overwritten when using
overwrite=true mode or throwing "The file already exists" errors when using overwrite=false
mode.
> We should switch to using a timestamp or some other unique generator to prevent filename
collisions regardless of the Producer instance lifecycle for the same hdfs directory URL...



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message