hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11920) Ozone : add key partition
Date Tue, 25 Jul 2017 13:51:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100063#comment-16100063
] 

Xiaoyu Yao commented on HDFS-11920:
-----------------------------------

Thanks [~vagarychen] for working on this. The patch looks good to me overall. A few comments
below

KeySpaceManagerProtocol.proto
Line 228: NIT: SubKeyInfo -> KeyLocation
Line 232: NIT: subeyDataOffset -> offset
Line 232: NIT: subeyDataSize -> length
Line 242: repeated KeyLocation locations

KSMSubKeyInfo.java
KsmSubKeyInfo -> KsmKeyLocation

OzoneClientImpl.java
Line 473: Do we need to create all the ChunkOutputstreams upon createKey? The other option
is to do 
lazy creation upon read/write to different (offset) containers.

Similar logic in DistributedStorageHandler can be consolidated with a shared utility functions.

OzoneKey.java
Line 41-48: we remove the single containerName/blockID from OzoneKey. Do we need to add the
list of 
containers/blocks info like we do for the KsmKeyInfo?

ChunkGroupOutputStream.java
Line 30-31: we could add some state into the block write state so that readers can detect
write failures.

Line 73/118: the ChunkOutputStreamEntry can be created as needed here.

Line 132: can we flush only the containers that is being written based on the state of blocks.

> Ozone : add key partition
> -------------------------
>
>                 Key: HDFS-11920
>                 URL: https://issues.apache.org/jira/browse/HDFS-11920
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Chen Liang
>            Assignee: Chen Liang
>         Attachments: HDFS-11920-HDFS-7240.001.patch, HDFS-11920-HDFS-7240.002.patch,
HDFS-11920-HDFS-7240.003.patch, HDFS-11920-HDFS-7240.004.patch
>
>
> Currently, each key corresponds to one single SCM block, and putKey/getKey writes/reads
to this single SCM block. This works fine for keys with reasonably small data size. However
if the data is too huge, (e.g. not even fits into a single container), then we need to be
able to partition the key data into multiple blocks, each in one container. This JIRA changes
the key-related classes to support this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message