hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8019) Erasure Coding: erasure coding chunk buffer allocation and management
Date Tue, 05 May 2015 10:02:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528210#comment-14528210

Vinayakumar B commented on HDFS-8019:

for client side erasure encoding, chunk buffers are obtained from {{ByteArrayManager}} provided
by the DFSClient. This will allow client to re-use the byte arrays (onheap) if the byte arrays
are having length more than threshold \[128 default\].

This was introduced in HDFS-7276. [~drankye] and [~szetszwo],  Do you think re-using {{ByteArrayManager}}
in datanode side erasure recovery would be sufficient for this?

> Erasure Coding: erasure coding chunk buffer allocation and management
> ---------------------------------------------------------------------
>                 Key: HDFS-8019
>                 URL: https://issues.apache.org/jira/browse/HDFS-8019
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Vinayakumar B
> As a task of HDFS-7344, this is to come up a chunk buffer pool allocating and managing
coding chunk buffers, either based on on-heap or off-heap. Note this assumes some DataNodes
are powerful in computing and performing EC coding work, so better to have this dedicated
buffer pool and management.

This message was sent by Atlassian JIRA

View raw message