camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Willem Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-8431) Consume all files in aws S3 bucket where deleteAfterRead = false
Date Thu, 08 Oct 2015 02:01:26 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14947949#comment-14947949
] 

Willem Jiang commented on CAMEL-8431:
-------------------------------------

I just checked the code, if the filename is not null and deleteAfterRead is false, camel always
pulls the same file again and again.
And if there some misunderstanding  with the setMarker and getMarker of the ListObjectRequest,
so I reopen the issue and will commit a quick fix for it shortly.

> Consume all files in aws S3 bucket where deleteAfterRead = false
> ----------------------------------------------------------------
>
>                 Key: CAMEL-8431
>                 URL: https://issues.apache.org/jira/browse/CAMEL-8431
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-aws
>    Affects Versions: 2.14.0
>            Reporter: Yap Poh Soon
>            Assignee: Willem Jiang
>             Fix For: 2.13.4, 2.14.3, 2.15.0
>
>
> The current AWS S3Consumer class has a problem if user supply the parameter deleteAfterRead=false.
 The S3Consumer will always consume the same files over and over again with the size of getMaxMessagesPerPoll().

> After some code chasing, i think, the root cause is because the ListObjectsRequest does
not has the previous marker value. Hence, i wonder if we could do something like below:
> {quote}
> {color:blue}private string marker; // new line to define the marker string {color}
> ..
> ..
> ..
> ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
> listObjectsRequest.setBucketName(bucketName);
> listObjectsRequest.setPrefix(getConfiguration().getPrefix());
> listObjectsRequest.setMaxKeys(maxMessagesPerPoll);
> {color:blue}listObjectsRequest.setMarker(marker); // new line to set the marker  {color}
    
> ObjectListing listObjects = getAmazonS3Client().listObjects(listObjectsRequest);
> {color:blue}marker = listObjects.getMarker(); // where marker is track          
> {color}{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message