hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-1094) Performance testing infrastructure : Special handling for zero-filled chunks on the Datanode
Date Wed, 24 Jul 2019 19:34:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892115#comment-16892115
] 

Eric Yang commented on HDDS-1094:
---------------------------------

HDDS-1772 implements some tests that fill up datanode disk that might be useful here.

> Performance testing infrastructure : Special handling for zero-filled chunks on the Datanode
> --------------------------------------------------------------------------------------------
>
>                 Key: HDDS-1094
>                 URL: https://issues.apache.org/jira/browse/HDDS-1094
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>          Components: Ozone Datanode
>            Reporter: Supratim Deka
>            Priority: Major
>
> Goal:
> Make Ozone chunk Read/Write operations CPU/network bound for specially constructed performance
micro benchmarks.
> Remove disk bandwidth and latency constraints - running ozone data path against extreme
low-latency & high throughput storage will expose performance bottlenecks in the flow.
But low-latency storage(NVME flash drives, Storage class memory etc) is expensive and availability
is limited. Is there a workaround which achieves similar running conditions for the software
without actually having the low latency storage? At least for specially constructed datasets
-  for example zero-filled blocks (*not* zero-length blocks).
> Required characteristics of the solution:
> No changes in Ozone client, OM and SCM. Changes limited to Datanode, Minimal footprint
in datanode code.
> Possible High level Approach:
> The ChunkManager and ChunkUtils can enable writeChunk for zero-filled chunks to be dropped
without actually writing to the local filesystem. Similarly, if readChunk can construct a
zero-filled buffer without reading from the local filesystem whenever it detects a zero-filled
chunk. Specifics of how to detect and record a zero-filled chunk can be discussed on this
jira. Also discuss how to control this behaviour and make it available only for internal testing.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message