hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12838) Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation requests
Date Mon, 15 Jan 2018 14:06:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326258#comment-16326258
] 

Mukul Kumar Singh commented on HDFS-12838:
------------------------------------------

Hi [~anu], the biggest problem is that the allocated containers are timing out after allocation.
This happens because the block cache delays the time for actual container creation, hence
SCM deletes the container after lease for the container expires. I have raised HDFS-12900
to fix this issue.

> Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation
requests
> -----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12838
>                 URL: https://issues.apache.org/jira/browse/HDFS-12838
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>            Priority: Major
>             Fix For: HDFS-7240
>
>         Attachments: HDFS-12838-HDFS-7240.001.patch, HDFS-12838-HDFS-7240.002.patch,
HDFS-12838-HDFS-7240.003.patch, HDFS-12838-HDFS-7240.004.patch
>
>
> Currently KeySpaceManager allocates multiple blocks by sending multiple block allocation
requests over the RPC. This can be optimized to aggregate multiple block allocation request
over one rpc.
> {code}
>       while (requestedSize > 0) {
>         long allocateSize = Math.min(scmBlockSize, requestedSize);
>         AllocatedBlock allocatedBlock =
>             scmBlockClient.allocateBlock(allocateSize, type, factor);
>         KsmKeyLocationInfo subKeyInfo = new KsmKeyLocationInfo.Builder()
>             .setContainerName(allocatedBlock.getPipeline().getContainerName())
>             .setBlockID(allocatedBlock.getKey())
>             .setShouldCreateContainer(allocatedBlock.getCreateContainer())
>             .setIndex(idx++)
>             .setLength(allocateSize)
>             .setOffset(0)
>             .build();
>         locations.add(subKeyInfo);
>         requestedSize -= allocateSize;
>       }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message