hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
Date Fri, 18 Aug 2017 08:11:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131887#comment-16131887
] 

Rakesh R commented on HDFS-12225:
---------------------------------

I thought to use the count for recursive traversal logic, I agree for this jira we don't need
this field. Below is the sample obj representation.
{code}
  static class SatisfyTrackInfo {
     private final long trackId;
     private final long rootNodeId;
    
     // Returns true if the tracking path is a directory, false otherwise.
     boolean isDir(){
           // add logic to represents directory or a file.
     }
  }
{code}

I'm adding few minor comments, could you please take care this as well.
# Please add {{DatanodeProtocol.DNA_DROP_SPS_WORK_COMMAND}} to BPOfferService so that DN won't
take any action on standby heartbeat response.
{code}
BPOfferService.java

BPOfferService#processCommandFromStandby()
    case DatanodeProtocol.DNA_ERASURE_CODING_RECONSTRUCTION:
    case DatanodeProtocol.DNA_BLOCK_STORAGE_MOVEMENT:
    case DatanodeProtocol.DNA_DROP_SPS_WORK_COMMAND:
{code}
# Send {{false}} value to the {{blockManager#stopSPS(false)}} function call while transitioning
from active to standby. This is to ensure that all pending xattr will be cleared only when
disabling SPS feature.
{code}
FSNamesystem.java

  void stopActiveServices() {
    LOG.info("Stopping services started for active state");
    writeLock();
    try {
      if (blockManager != null) {
        blockManager.stopSPS(true);
      }
{code}
# How about clear sps queues during sps stop like,
{code}
StoragePolicySatisfier.java

StoragePolicySatisfier#disable(){
   //...
   //..
   // clear queue at the end
   clearQueues();
}
{code} 

> [SPS]: Optimize extended attributes for tracking SPS movements
> --------------------------------------------------------------
>
>                 Key: HDFS-12225
>                 URL: https://issues.apache.org/jira/browse/HDFS-12225
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Uma Maheswara Rao G
>            Assignee: Surendra Singh Lilhore
>         Attachments: HDFS-12225-HDFS-10285-01.patch, HDFS-12225-HDFS-10285-02.patch
>
>
> We have discussed to optimize number extended attributes and asked to report separate
JIRA while implementing [HDFS-11150 | https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work 
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only mark the
directory. When recovering from FsImage, the InodeMap isn't built up, so we don't know the
sub-inode of a given inode, in the end, We cannot add these inodes to movement queue in FSDirectory#addToInodeMap,
any thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can add for
all Inodes now. For this to handle 100%, we may need intermittent processing, like first we
should add them to some intermittentList while loading fsImage, once fully loaded and when
starting active services, we should process that list and do required stuff. But it would
add some additional complexity may be. Let's do with all file inodes now and we can revisit
later if it is really creating issues. How about you raise a JIRA for it and think to optimize
separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899 also the cursor
of the iterator in the EZ root xattr to track progress and handle restarts. I wonder if we
can do something similar here to avoid having an xattr-per-file being moved.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message