hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12838) Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation requests
Date Wed, 24 Jan 2018 20:35:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338178#comment-16338178

Anu Engineer commented on HDFS-12838:

[~msingh] Thanks for investigating the cache. This patch looks quite good. I have only one

* KeyManagerImpl.java:allocateBlocks*
 while (requestedSize > 0) {
      long allocateSize = Math.min(scmBlockSize, requestedSize);
      requestedSize -= allocateSize;

I am presuming that we are trying to do here it to convert
the requested size to blocks. For example, if the block size is 128M and user requested 500M
of space, we will convert that into 500MB / 128MB = 4 blocks, yielding an allocation in SCM
of 4 blocks.

If my assumption is true, then would it make sense to
rewrite the following API 
{{scmBlockClient.allocateBlock(sizes, type, factor, ksmId);}}
{{scmBlockClient.allocateBlock(blockSize, blockCount, type, factor, ksmId);}}
and now the original while loop can be replaced as 

int blockCount = (requestedSize % scmBlockSize) + 1;
scmBlockClient.allocateBlock(scmBlockSize, blockCount, type, factor, ksmId);

This does mean that client will have to overwrite the actual block length after the update
is done.

This means that the signature in {{ScmBlockLocationProtocol.java}} would have to change too
and ScmBlockLocationProcotol.proto -- would change to 
{{uint64 size = 1;}} and {{uint32 count = 2;}}

* nit: ScmBlockLocationProtocolClientSideTranslatorPB, seems like we have an unused import.

> Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation
> -----------------------------------------------------------------------------------------------
>                 Key: HDFS-12838
>                 URL: https://issues.apache.org/jira/browse/HDFS-12838
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>            Priority: Major
>             Fix For: HDFS-7240
>         Attachments: HDFS-12838-HDFS-7240.001.patch, HDFS-12838-HDFS-7240.002.patch,
HDFS-12838-HDFS-7240.003.patch, HDFS-12838-HDFS-7240.004.patch, HDFS-12838-HDFS-7240.005.patch,
> Currently KeySpaceManager allocates multiple blocks by sending multiple block allocation
requests over the RPC. This can be optimized to aggregate multiple block allocation request
over one rpc.
> {code}
>       while (requestedSize > 0) {
>         long allocateSize = Math.min(scmBlockSize, requestedSize);
>         AllocatedBlock allocatedBlock =
>             scmBlockClient.allocateBlock(allocateSize, type, factor);
>         KsmKeyLocationInfo subKeyInfo = new KsmKeyLocationInfo.Builder()
>             .setContainerName(allocatedBlock.getPipeline().getContainerName())
>             .setBlockID(allocatedBlock.getKey())
>             .setShouldCreateContainer(allocatedBlock.getCreateContainer())
>             .setIndex(idx++)
>             .setLength(allocateSize)
>             .setOffset(0)
>             .build();
>         locations.add(subKeyInfo);
>         requestedSize -= allocateSize;
>       }
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message