hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-13762) Support non-volatile storage class memory(SCM) in HDFS cache directives
Date Tue, 05 Feb 2019 04:47:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760449#comment-16760449
] 

Rakesh R edited comment on HDFS-13762 at 2/5/19 4:46 AM:
---------------------------------------------------------

Thanks [~PhiloHe] for the continuous efforts!
{quote}Yes, it's a limitation for all the volumes together.
{quote}
OK, it would be concern when there are different sizes volumes. Agreed to take up in a follow
up jira task.
{quote}We maintains the map from block file to cache-file on pmem storage, so no need to search
for it among the volumes
{quote}
Great!
{quote}We don't have a evict logic here or throw a VolumeFullException, it just works like
the memory-cache for compatible now.
{quote}
Please add this case to the system test plan. It would be good to test it and document/know the
behavior.
{quote}(iii), we will update the patch accordingly
{quote}
Please change below one also.
{code:java}
+          Pmem.unmapBlock(region.getAddress(), region.getLength());
+          boolean deled = false;
{code}
 

Adding few more comments, please take care:

1). MappableBlock interface looks good, please add javadocs for the below functions. Also,
please reflect specific class MappableBlock, PmemMappedBlock, MemoryMappedBlock responsibility
in javadoc instead of using the same "Represents an HDFS block that is mmapped by the DataNode."
{code:java}
  long getLength();

  void afterCache();
{code}
2). Change the comment to PMDK instead of ISA-L!
{code:java}
+  // Load Intel ISA-L
+  #ifdef UNIX
{code}
3). Could you please brief the difference between {{pmemDrain}} and {{pmemSync}}. Also, I
would appreciate if you could add javadoc so that the functionality will be visible to the
readers.
{code:java}
    private static native boolean isPmemCheck(long address, long length);
    private static native PmemMappedRegion pmemCreateMapFile(String path,
        long length);
    private static native boolean pmemUnMap(long address, long length);
    private static native void pmemCopy(byte[] src, long dest, boolean isPmem,
        long length);
    private static native void pmemDrain();
    private static native void pmemSync(long address, long length);
{code}


was (Author: rakeshr):
Thanks [~PhiloHe] for the continuous efforts!

{quote}Yes, it's a limitation for all the volumes together.
{quote}
OK, it would be concern when there are different sizes volumes. Agreed to take up in a follow
up jira task.
{quote}We maintains the map from block file to cache-file on pmem storage, so no need to search
for it among the volumes
{quote}
Great!
{quote}We don't have a evict logic here or throw a VolumeFullException, it just works like
the memory-cache for compatible now.
{quote}
Please add this case to the system test plan. It would be good to test it and document/know the
behavior.
{quote}(iii), we will update the patch accordingly
{quote}
Please change below one also.
{code:java}
+          Pmem.unmapBlock(region.getAddress(), region.getLength());
+          boolean deled = false;
{code}
 

Adding few more comments, please take care:
 # MappableBlock interface looks good, please add javadocs for the below functions. Also,
please reflect specific class MappableBlock, PmemMappedBlock, MemoryMappedBlock responsibility
in javadoc instead of using the same "Represents an HDFS block that is mmapped by the DataNode."
{code:java}
  long getLength();

  void afterCache();
{code}

 # Change the comment to PMDK instead of ISA-L!
{code:java}
+  // Load Intel ISA-L
+  #ifdef UNIX
{code}

 # Could you please brief the difference between {{pmemDrain}} and {{pmemSync}}. Also, I would
appreciate if you could add javadoc so that the functionality will be visible to the readers.
{code:java}
    private static native boolean isPmemCheck(long address, long length);
    private static native PmemMappedRegion pmemCreateMapFile(String path,
        long length);
    private static native boolean pmemUnMap(long address, long length);
    private static native void pmemCopy(byte[] src, long dest, boolean isPmem,
        long length);
    private static native void pmemDrain();
    private static native void pmemSync(long address, long length);
{code}

> Support non-volatile storage class memory(SCM) in HDFS cache directives
> -----------------------------------------------------------------------
>
>                 Key: HDFS-13762
>                 URL: https://issues.apache.org/jira/browse/HDFS-13762
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: caching, datanode
>            Reporter: Sammi Chen
>            Assignee: Feilong He
>            Priority: Major
>         Attachments: HDFS-13762.000.patch, HDFS-13762.001.patch, HDFS-13762.002.patch,
HDFS-13762.003.patch, HDFS-13762.004.patch, HDFS-13762.005.patch, HDFS-13762.006.patch, HDFS-13762.007.patch,
HDFS-13762.008.patch, SCMCacheDesign-2018-11-08.pdf, SCMCacheTestPlan.pdf
>
>
> No-volatile storage class memory is a type of memory that can keep the data content after
power failure or between the power cycle. Non-volatile storage class memory device usually
has near access speed as memory DIMM while has lower cost than memory.  So today It is usually
used as a supplement to memory to hold long tern persistent data, such as data in cache. 
> Currently in HDFS, we have OS page cache backed read only cache and RAMDISK based lazy
write cache.  Non-volatile memory suits for both these functions. 
> This Jira aims to enable storage class memory first in read cache. Although storage class
memory has non-volatile characteristics, to keep the same behavior as current read only cache,
we don't use its persistent characteristics currently.  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message