hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuanbo Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
Date Wed, 16 Aug 2017 07:19:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128428#comment-16128428
] 

Yuanbo Liu commented on HDFS-12283:
-----------------------------------

[~cheersyang] Thanks a lot for your review.
Agree with the most part of your comments and I've addressed them in v2 patch.
{quote}
line 110: this might have problem.. e.g you have listed tx 1,3,5 (2,4 has been committed),
then 4 won't be a valid start key anymore.
{quote}
It's quite a tricky part because committing transactions could happen in any time. So we decide
to use a filter instead of  a startKey to avoid repeatedly scanning the same records. The
whole idea is based on the sequentially increasing txid,  we record the last read txid, scan
the db from the very beginning, and get the transactions whose id are larger than the last
read txid, so that the invalid start key issue could be addressed.

> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -------------------------------------------------
>
>                 Key: HDFS-12283
>                 URL: https://issues.apache.org/jira/browse/HDFS-12283
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone, scm
>            Reporter: Weiwei Yang
>            Assignee: Yuanbo Liu
>         Attachments: HDFS-12283.001.patch, HDFS-12283-HDFS-7240.001.patch, HDFS-12283-HDFS-7240.002.patch
>
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container blocks which
are under deletion. It maintains info about under-deletion container blocks that notified
by KSM, and the state how it is processed. We can use RocksDB to implement the 1st version
of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode, it represents
the "state" of the transaction, it is in range of \[-1, 5\], -1 means the transaction eventually
failed after some retries, 5 is the max number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with RocksDB
{{MetadataStore}} as the first version.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message