nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Skora <jsk...@gmail.com>
Subject Re: PutS3Object submitting file many times
Date Wed, 26 Oct 2016 16:53:18 GMT
I agree with Joe, it sounds like the NiFi commit was failing after the file
was sent.

If the processor was administratively yielding, possibly due to the disk
issues you mentioned, after completing the upload to S3, the file would be
left in the queue and picked up again for processing later, creating the
situation you observed.


On Wed, Oct 26, 2016 at 3:13 PM, Joe Witt <joe.witt@gmail.com> wrote:

> Joe
>
> I have not looked at this code but can say that processors are or should be
> built such that in error cases we prefer duplication over loss.  So a case
> like you describe could be a good example where session commit fails or
> something like it causing retries but in the mean time we keep sending to
> the destination.  It generally can be avoided that it would be so
> pronounced but just as a heads up I thought id call this out.
>
> Thanks
> Joe
>
> On Oct 26, 2016 9:58 AM, "Joe Gresock" <jgresock@gmail.com> wrote:
>
> > Has anyone run into problems where PutS3Object submits the same file many
> > (hundreds) of times?  This turned out to be a problem for me since we
> have
> > bucket versioning enabled, so we have many copies of the same file.
> >
> > I can't reproduce this behavior anymore in the normal flow of things, but
> > I'm wondering if it's possible for this to happen if the processor is
> > forced to administratively yield (i.e., due to disk full or too many open
> > files).
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.    *-Philippians 4:12-13*
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message