nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mans Singh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (NIFI-1325) Enhance AWS S3 fetch to access bucket across accounts
Date Mon, 28 Dec 2015 22:00:50 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073186#comment-15073186
] 

Mans Singh edited comment on NIFI-1325 at 12/28/15 10:00 PM:
-------------------------------------------------------------

@tkurc - I am looking at the AWS credentials api and it looks like there is an interface -
AWSCredentialsProvider which as various types of credential providers - (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AWSCredentialsProvider.html).
 The interface has a refresh method.

Currently, the Nifi AbstractAWSProcessor has a method     

protected abstract ClientType createClient(final ProcessContext context, final AWSCredentials
credentials,
            final ClientConfiguration config);

If we can change it to 

protected abstract ClientType createClient(final ProcessContext context, final AWSCredentialsProvider
credentialsProvider,
            final ClientConfiguration config);

Then we change the createClient method in AbstractS3Processor to return AmazonS3Client with
the creds provider rather than with the creds (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html).
 Internally, when created with creds the AmazonS3Client creates a static credentials provider
but it can also be instantiated with creds provider (in our case, like http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html).

There is are two other processor AbstractSNSProcessor and AbstractSQSProcessor in the NIFi
Aws components and they can also be refactored to use the creds provider (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sns/AmazonSNSClient.html,
and http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html).

There is only one issue - the AbstractAWSProcessor.getCredentails method (which will change
to getCredentialsProvider) returns AnonymousAWSCredentials for which I could not find a credentials
provider, but we can always add one if required.
 
This looks like a much more flexible and extensible solution even for other aws components.


Please let me know your thoughts/recommendations.





was (Author: mans2singh):
@tkurc - I am looking at the AWS credentials api and it looks like there is an interface -
AWSCredentialsProvider which as various types of credential providers - (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AWSCredentialsProvider.html).
 The interface has a refresh method.

Currently, the Nifi AbstractAWSProcessor has a method     

protected abstract ClientType createClient(final ProcessContext context, final AWSCredentials
credentials,
            final ClientConfiguration config);

If we can change it to 

protected abstract ClientType createClient(final ProcessContext context, final AWSCredentialsProvider
credentialsProvider,
            final ClientConfiguration config);

Then we change the createClient method in AbstractS3Processor to return AmazonS3Client with
the creds provider rather than with the creds (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html).
 Internally, when created with creds the AmazonS3Client creates a static credentials provider
but it can also be instantiated with creds provider (in our case, like http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html).

There is are two other processor AbstractSNSProcessor and AbstractSQSProcessor in the NIFi
Aws components and it can also be refactored to use the creds provider (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sns/AmazonSNSClient.html,
and http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html).

There is only one issue - the AbstractAWSProcessor.getCredentails method (which will change
to getCredentialsProvider) returns AnonymousAWSCredentials for which I could not find a credentials
provider, but we can always add one if required.
 
This looks like a much more flexible and extensible solution even for other aws components.


Please let me know your thoughts/recommendations.




> Enhance AWS S3 fetch to access bucket across accounts
> -----------------------------------------------------
>
>                 Key: NIFI-1325
>                 URL: https://issues.apache.org/jira/browse/NIFI-1325
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 0.4.1
>         Environment: All
>            Reporter: Mans Singh
>            Assignee: Tony Kurc
>            Priority: Minor
>              Labels: easyfix
>             Fix For: 0.4.1
>
>         Attachments: nifi-1325.patch.zip
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> The AWS S3 Fetch Object component does not allow access to bucket across accounts. AWS
 S3 Fetch Object with can be enhanced to provide this functionality by using assume role session/credentials




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message