spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian London (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-10969) Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
Date Wed, 02 Dec 2015 15:32:11 GMT

    [ https://issues.apache.org/jira/browse/SPARK-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035963#comment-15035963
] 

Brian London edited comment on SPARK-10969 at 12/2/15 3:31 PM:
---------------------------------------------------------------

I'm not sure why it needs to be an object as opposed to two strings, or is the issue that
this change only adds custom credentials to kinesis and not dynamodb and cloudwatch?  

There are already the `fs.s3.awsAccessKeyId` and `fs.s3.awsSecretAccessKey` parameters that
are used for the `.saveToTextFile` RDD method.  Perhaps something like that could be used
for the other services as well.


was (Author: brianlondon):
I'm not sure why it needs to be an object as opposed to two strings, or is the issue that
this change only adds custom credentials to kinesis and not dynamodb and cloudwatch?  There
are already the `fs.s3.awsAccessKeyId` and `fs.s3.awsSecretAccessKey` parameters that are
used for the `.saveToTextFile` RDD method.  Perhaps something like that could be used for
the other services as well.

> Spark Streaming Kinesis: Allow specifying separate credentials for Kinesis and DynamoDB
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-10969
>                 URL: https://issues.apache.org/jira/browse/SPARK-10969
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.5.1
>            Reporter: Christoph Pirkl
>            Priority: Critical
>
> {{KinesisUtils.createStream()}} allows specifying only one set of AWS credentials that
will be used by Amazon KCL for accessing Kinesis, DynamoDB and CloudWatch.
> h5. Motivation
> In a scenario where one needs to read from a Kinesis Stream owned by a different AWS
account the user usually has minimal rights (i.e. only read from the stream). In this case
creating the DynamoDB table in KCL will fail.
> h5. Proposal
> My proposed solution would be to allow specifying multiple credentials in {{KinesisUtils.createStream()}}
for Kinesis, DynamoDB and CloudWatch. The additional credentials could then be passed to the
constructor of {{KinesisClientLibConfiguration}} or method {{KinesisClientLibConfiguration.withDynamoDBClientConfig()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message