hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HDFS-12777) [READ] Reduce memory and CPU footprint for PROVIDED volumes.
Date Fri, 10 Nov 2017 01:54:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246894#comment-16246894

Íñigo Goiri commented on HDFS-12777:

[^HDFS-12777-HDFS-9806.004.patch] should be fine for checkstyle.
The unit test in 003 seem unrelated.
Wait for 004 build from Jenkins but other than that +1.

> [READ] Reduce memory and CPU footprint for PROVIDED volumes.
> ------------------------------------------------------------
>                 Key: HDFS-12777
>                 URL: https://issues.apache.org/jira/browse/HDFS-12777
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Virajith Jalaparti
>            Assignee: Virajith Jalaparti
>         Attachments: HDFS-12777-HDFS-9806.001.patch, HDFS-12777-HDFS-9806.002.patch,
HDFS-12777-HDFS-9806.003.patch, HDFS-12777-HDFS-9806.004.patch
> As opposed to local blocks, each DN keeps track of all blocks in PROVIDED storage. This
can be millions of blocks for 100s of TBs of PROVIDED data. Storing the data for these blocks
can lead to a large memory footprint. Further, with so many blocks, {{DirectoryScanner}} running
on a PROVIDED volume can increase the memory and CPU utilization. 
> To reduce these overheads, this JIRA aims to (a) disable the {{DirectoryScanner}} on
PROVIDED volumes (as HDFS-9806 focuses on only read-only data in PROVIDED volumes), (b) reduce
the space occupied by {{FinalizedProvidedReplicaInfo}} by using a common URI prefix across
all PROVIDED blocks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message