hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sameer Choudhary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
Date Thu, 09 Feb 2017 00:23:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858755#comment-15858755

Sameer Choudhary commented on HADOOP-13345:


Today, I attended the talk on the project at Spark Summit 2017. Thanks for putting in all
the effort!

I have a question regarding pricing of DynamoDB. It charges on read/write request rate. So,
users might have to pay high amount of price for getting the consistency guarantees. This
would especially affect large Spark Jobs with many parallel executing tasks that are trying
to read/write to DynamoDB. Putting throttling will affect the job performance. Some benchmarks
here would be great.

A solution could be for S3Guard to additionally support for custom Key Value store such as
Apache HBase that supports strictly consistent reads/writes. A user can create a separate
cluster or use the same Spark cluster to setup the store. The benefit of the approach is that
users can now achieve high throughput on even large Spark jobs with paying just a fraction
of cost.

> S3Guard: Improved Consistency for S3A
> -------------------------------------
>                 Key: HADOOP-13345
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13345
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch, S3C-ConsistentListingonS3-Design.pdf,
S3GuardImprovedConsistencyforS3A.pdf, S3GuardImprovedConsistencyforS3AV2.pdf
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a stronger
consistency model than what is currently offered.  The solution coordinates with a strongly
consistent external store to resolve inconsistencies caused by the S3 eventual consistency

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message