hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Georgi Chalakov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14520) Block compaction for WASB (Block Blobs Instead of Page Plobs)
Date Sun, 09 Jul 2017 08:19:02 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Georgi Chalakov updated HADOOP-14520:
-------------------------------------
    Release Note: Block Compaction for WASB. When the number of blocks in a block blob is
above 32000, the process of compaction replaces a sequence of small blocks with with one big
block.   (was: Block Compaction for WASB. When the number of blocks in a block blob is above
32000, compaction replaces longest sequence of blocks with total size length less then 4M,
with just one block. Compaction allows blocks blobs to be used instead of page blobs, including
for WAL files.)
          Status: Patch Available  (was: In Progress)

> Block compaction for WASB (Block Blobs Instead of Page Plobs)
> -------------------------------------------------------------
>
>                 Key: HADOOP-14520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14520
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/azure
>    Affects Versions: 3.0.0-alpha3
>            Reporter: Georgi Chalakov
>            Assignee: Georgi Chalakov
>         Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt, HADOOP-14520-03.patch,
HADOOP-14520-4.patch
>
>
> Block Compaction for WASB allows uploading new blocks for every hflush/hsync call. When
the number of blocks is above 32000, next hflush/hsync triggers the block compaction process.
Block compaction replaces a sequence of blocks with one block. From all the sequences with
total length less than 4M, compaction chooses the longest one. It is a greedy algorithm that
preserve all potential candidates for the next round. Block Compaction for WASB increases
data durability and allows using block blobs instead of page blobs. By default, block compaction
is disabled. Similar to the configuration for page blobs, the client needs to specify HDFS
folders where block compaction over block blobs is enabled. 
> Results for HADOOP-14520-04.patch
> tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net
> Tests run: 701, Failures: 0, Errors: 0, Skipped: 119



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message