flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Norris <ffrank7...@mailbox.org>
Subject Re: Flume stops writing data to s3
Date Wed, 07 Dec 2016 09:03:03 GMT
Log and cry are as below:

http://paste.debian.net/901028

https://paste.debian.net/901029

Actually there is no error or exception thrown. The problem is when observing 204 then flume
stops consuming(only keep printing 'checkpoint not required' msg to log), resulting in the
records in source and sink are not the same. This usually happens with consuming more than
15k records. When the number of records are not big e.g. 5k everything works fine (after moving
data from source to sink, the record number is the same). And at this moment no transformation,
etc. involves, just plain moving stuff from source to sink.

Thanks for help.


> On 05 December 2016 at 22:36 iain wright <iainwrig@gmail.com> wrote:
> 
>     seems http 204 is exepected response from S3 delete, can you pastebin your log/cfg
(please omit keys if present in cfg)?
> 
>     Does your sink stop consuming from the channel when that error is encountered? 
> 
> 
>     --
>     Iain Wright
> 
>     This email message is confidential, intended only for the recipient(s) named above
and may contain information that is privileged, exempt from disclosure under applicable law.
If you are not the intended recipient, do not disclose or disseminate the message to anyone
except the intended recipient. If you have received this message in error, or are not the
named recipient(s), please immediately notify the sender by return email, and delete all copies
of this message.
> 
>     On Mon, Dec 5, 2016 at 12:27 PM, Frank Norris <ffrank7482@mailbox.org mailto:ffrank7482@mailbox.org
> wrote:
> 
>         > > 
> >         I use flume v1.6 and write data to s3 sink. However flume stops processing
(no crash still print log message) when it receives http code 204. In log it shows flume was
deleting files. Any places I can check to avoid this problem or further investigation?
> > 
> >         Thanks
> > 
> >     > 
> 

Mime
View raw message