nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "endzeit (Jira)" <j...@apache.org>
Subject [jira] [Updated] (NIFI-7371) Improve error handling of S3 processors
Date Thu, 16 Apr 2020 23:15:00 GMT

     [ https://issues.apache.org/jira/browse/NIFI-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

endzeit updated NIFI-7371:
--------------------------
    Description: 
Currently the S3 processors, such as FetchS3Object, only expose the relationsships "success"
and "failure". However the are a multitute of reasons why an interaction with an S3 storage
might fail.

As of now there is no easy way of knowing why a flow file was directed to the "failure" relationsship
just by the flow file itself.

A way of finding out the reason might be to search for a corresponing log / bulletin entry.
 This seems rather complicated.

A use case where having this information is useful could be when deciding whether
 * the action should be retried, e.g. on a timeout,
 * or the failure be handled, e.g. when the object for the specified key does not exists.

I haven't looked much into the underlying AWS library yet as I wanted to discuss whether such
an improvement is desired at all first?
 If so, should the information be exposed via
 * additional / other relationsships, such as in the FetchSFTP processor,
 * or rather an attribute added to the flow file, such as in the ValidateXml processor?

This might also apply to other AWS processors but we just have come across the S3 processors
as we use them quite regulary.

Any thoughts or comments would be highly appreciated!

  was:
Currently the S3 processors, such as FetchS3Object, only expose the relationsships "success"
and "failure". However the are a multitute of reasons why an interaction with an S3 storage
might fail.

As of now there is no easy way of knowing why a flow file was directed to the "failure" relationsship
just by the flow file itself.

A way of finding out the reason might be to search for a corresponing log / bulletin entry.
This seems rather complicated.

A use case where having this information is useful could be when deciding whether 
 * the action should be retried, e.g. on a timeout,
 * or the failure be handled, e.g. when the object for the specified key does not exists.

I haven't looked much into the underlying AWS library yet as I wanted to discuss whether such
an improvement is desired at all first?
If so, should the information be exposed via 
 * additional / other relationsships, such as in the FetchSFTP processor, 
 * or rather an attribute added to the flow file, such as in the ValidateXml processor?

This might also apply to other AWS processors but we jut have come across the S3 processors
as we use them quite regulary.

Any thoughts or comments would be highly appreciated!


> Improve error handling of S3 processors
> ---------------------------------------
>
>                 Key: NIFI-7371
>                 URL: https://issues.apache.org/jira/browse/NIFI-7371
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.11.4
>            Reporter: endzeit
>            Priority: Major
>
> Currently the S3 processors, such as FetchS3Object, only expose the relationsships "success"
and "failure". However the are a multitute of reasons why an interaction with an S3 storage
might fail.
> As of now there is no easy way of knowing why a flow file was directed to the "failure"
relationsship just by the flow file itself.
> A way of finding out the reason might be to search for a corresponing log / bulletin
entry.
>  This seems rather complicated.
> A use case where having this information is useful could be when deciding whether
>  * the action should be retried, e.g. on a timeout,
>  * or the failure be handled, e.g. when the object for the specified key does not exists.
> I haven't looked much into the underlying AWS library yet as I wanted to discuss whether
such an improvement is desired at all first?
>  If so, should the information be exposed via
>  * additional / other relationsships, such as in the FetchSFTP processor,
>  * or rather an attribute added to the flow file, such as in the ValidateXml processor?
> This might also apply to other AWS processors but we just have come across the S3 processors
as we use them quite regulary.
> Any thoughts or comments would be highly appreciated!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message