hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] tomscut opened a new pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
Date Thu, 19 Aug 2021 03:54:51 GMT

tomscut opened a new pull request #3313:
URL: https://github.com/apache/hadoop/pull/3313


   JIRA: [HDFS-16179](https://issues.apache.org/jira/browse/HDFS-16179)
   
   ```
   private void chooseExcessRedundancyStriped(BlockCollection bc,
       final Collection<DatanodeStorageInfo> nonExcess,
       BlockInfo storedBlock,
       DatanodeDescriptor delNodeHint) {
     ...
     // cardinality of found indicates the expected number of internal blocks
     final int numOfTarget = found.cardinality();
     final BlockStoragePolicy storagePolicy = storagePolicySuite.getPolicy(
         bc.getStoragePolicyID());
     final List<StorageType> excessTypes = storagePolicy.chooseExcess(
         (short) numOfTarget, DatanodeStorageInfo.toStorageTypes(nonExcess));
     if (excessTypes.isEmpty()) {
       LOG.warn("excess types chosen for block {} among storages {} is empty",
           storedBlock, nonExcess);
       return;
     }
     ...
   }
   ```
   
   IMO, here is just detecting excess StorageType and setting the log level to debug has no
effect.
    
   We have a cluster that uses the EC policy to store data. The current log level is WARN
here, and in about 50 minutes, 286,093 logs are printed, which can cause other important logs
to drown out.
   
   ![image](https://user-images.githubusercontent.com/55134131/130005007-ea4cd5d5-f18a-4a06-855e-ba1c98a93fcb.png)
   
   ![image](https://user-images.githubusercontent.com/55134131/130005037-57bc2116-68de-48cd-a81c-52f866336751.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message