hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
Date Thu, 26 Jan 2017 07:12:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839356#comment-15839356
] 

Aaron Fabbri commented on HADOOP-13936:
---------------------------------------

This is part of a general class of problems.  In general, since we do not have transactions
around updating the two sources of truth, things can get out of sync when failures happen.

Transactions are probably not worth the cost here.  The biggest issue IMO is the fact that
we're running in the FS client.  You really want a long-lived daemon with durable storage
to implement transactions.  Another significant issue is the runtime cost in terms of more
round trips.

The current approach is to detect problems when they occur and give folks tools to repair
them.  Simply wiping the MetadataStore is one easy way to fix problems.  Another more automatic,
but still simple approach, that I plan on implementing in the future, is to set configurable
TTL on entries in the MetadataStore.  We will contribute a patch soon which adds a "expire"
or "trim" command to the CLI, which will delete any entries older than X hours.  We could
also add a config knob to the client which ignores entries past a certain TTL.    That doesn't
immediately fix inconsistencies, but does cause them to automatically heal after some time.

> S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-13936
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13936
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>
> As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked, which deletes
keys from S3 in batches (default is 1000). But DynamoDB is updated only at the end of this
operation. This can cause issues when deleting large number of keys. 
> E.g, it is possible to get exception after deleting 1000 keys and in such cases dynamoDB
would not be updated. This can cause DynamoDB to go out of sync. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message