hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-1094) Performance test infrastructure : skip writing user data on Datanode
Date Wed, 21 Aug 2019 19:52:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912634#comment-16912634
] 

Anu Engineer commented on HDDS-1094:
------------------------------------

[~sdeka] Would it be not be easier to add this skip in the client itself, so that data is
never pushed at all ?

 

> Performance test infrastructure : skip writing user data on Datanode
> --------------------------------------------------------------------
>
>                 Key: HDDS-1094
>                 URL: https://issues.apache.org/jira/browse/HDDS-1094
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>          Components: Ozone Datanode
>            Reporter: Supratim Deka
>            Assignee: Supratim Deka
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Goal:
> Make Ozone chunk Read/Write operations CPU/network bound for specially constructed performance
micro benchmarks.
> Remove disk bandwidth and latency constraints - running ozone data path against extreme
low-latency & high throughput storage will expose performance bottlenecks in the flow.
But low-latency storage(NVME flash drives, Storage class memory etc) is expensive and availability
is limited. Is there a workaround which achieves similar running conditions for the software
without actually having the low latency storage? At least for specially constructed datasets
-  for example zero-filled blocks (*not* zero-length blocks).
> Required characteristics of the solution:
> No changes in Ozone client, OM and SCM. Changes limited to Datanode, Minimal footprint
in datanode code.
> Possible High level Approach:
> The ChunkManager and ChunkUtils can enable writeChunk for zero-filled chunks to be dropped
without actually writing to the local filesystem. Similarly, if readChunk can construct a
zero-filled buffer without reading from the local filesystem whenever it detects a zero-filled
chunk. Specifics of how to detect and record a zero-filled chunk can be discussed on this
jira. Also discuss how to control this behaviour and make it available only for internal testing.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message