hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewan Higgs (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-12665) [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb)
Date Mon, 30 Oct 2017 15:45:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ewan Higgs updated HDFS-12665:
------------------------------
    Attachment: HDFS-12665-HDFS-9806.003.patch

Attaching patch that rebased on the latest HDFS-11902 patch. This also takes into account
many of [~virajith]'s comments. 

One thing to note: I kept ITAliasMap because it attempts to use the same port as some of the
unit tests and they collide when run in parallel. By keeping these tests in an integration
test they no longer compete for ports.

> [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb)
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-12665
>                 URL: https://issues.apache.org/jira/browse/HDFS-12665
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ewan Higgs
>            Assignee: Ewan Higgs
>         Attachments: HDFS-12665-HDFS-9806.001.patch, HDFS-12665-HDFS-9806.002.patch,
HDFS-12665-HDFS-9806.003.patch
>
>
> The design of Provided Storage requires the use of an AliasMap to manage the mapping
between blocks of files on the local HDFS and ranges of files on a remote storage system.
To reduce load from the Namenode, this can be done using a pluggable external service (e.g.
AzureTable, Cassandra, Ratis). However, to aide adoption and ease of deployment, we propose
an in memory version.
> This AliasMap will be a wrapper around LevelDB (already a dependency from the Timeline
Service) and use protobuf for the key (blockpool, blockid, and genstamp) and the value (url,
offset, length, nonce). The in memory service will also have a configurable port on which
it will listen for updates from Storage Policy Satisfier (SPS) Coordinating Datanodes (C-DN).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message