hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-14579) In refreshNodes, avoid performing a DNS lookup while holding the write lock
Date Tue, 18 Jun 2019 20:23:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867011#comment-16867011
] 

Íñigo Goiri edited comment on HDFS-14579 at 6/18/19 8:22 PM:
-------------------------------------------------------------

So here you can see this stuck at getting the address:
{code}
"main" #1 prio=5 os_prio=0 tid=0x0000000001b38800 nid=0x6a0c runnable [0x0000000001b1e000]
   java.lang.Thread.State: RUNNABLE
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at java.net.InetAddress.getByName(InetAddress.java:1076)
	at java.net.InetSocketAddress.<init>(InetSocketAddress.java:220)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.parseEntry(HostFileManager.java:94)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.readFile(HostFileManager.java:80)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:157)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:70)
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.<init>(DatanodeManager.java:274)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:416)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:408)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:792)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:673)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:750)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1000)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:979)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1726)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1794)
{code}

The code path is not exactly the same but it's the same DNS resolution issue.
This is Hadoop 2.9.1 BTW.


was (Author: elgoiri):
So here you can see this stuck at getting the address:
{code}
"main" #1 prio=5 os_prio=0 tid=0x0000000001b38800 nid=0x6a0c runnable [0x0000000001b1e000]
   java.lang.Thread.State: RUNNABLE
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at java.net.InetAddress.getByName(InetAddress.java:1076)
	at java.net.InetSocketAddress.<init>(InetSocketAddress.java:220)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.parseEntry(HostFileManager.java:94)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.readFile(HostFileManager.java:80)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:157)
	at org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager.refresh(HostFileManager.java:70)
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.<init>(DatanodeManager.java:274)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:416)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:408)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:792)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:673)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:750)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1000)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:979)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1726)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1794)
{code}

The code path is not exactly the same but it's the same DNS resolution issue.

> In refreshNodes, avoid performing a DNS lookup while holding the write lock
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-14579
>                 URL: https://issues.apache.org/jira/browse/HDFS-14579
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 3.3.0
>            Reporter: Stephen O'Donnell
>            Assignee: Stephen O'Donnell
>            Priority: Major
>         Attachments: HDFS-14579.001.patch
>
>
> When refreshNodes is called on a large cluster, or a cluster where DNS is not performing
well, it can cause the namenode to hang for a long time. This is because the refreshNodes
operation holds the global write lock while it is running. Most of refreshNodes code is simple
and hence fast, but unfortunately it performs a DNS lookup for each host in the cluster while
the lock is held. 
> Right now, it calls:
> {code}
>   public void refreshNodes(final Configuration conf) throws IOException {
>     refreshHostsReader(conf);
>     namesystem.writeLock();
>     try {
>       refreshDatanodes();
>       countSoftwareVersions();
>     } finally {
>       namesystem.writeUnlock();
>     }
>   }
> {code}
> The line refreshHostsReader(conf); reads the new config file and does a DNS lookup on
each entry - the write lock is not held here. Then the main work is done here:
> {code}
>   private void refreshDatanodes() {
>     final Map<String, DatanodeDescriptor> copy;
>     synchronized (this) {
>       copy = new HashMap<>(datanodeMap);
>     }
>     for (DatanodeDescriptor node : copy.values()) {
>       // Check if not include.
>       if (!hostConfigManager.isIncluded(node)) {
>         node.setDisallowed(true);
>       } else {
>         long maintenanceExpireTimeInMS =
>             hostConfigManager.getMaintenanceExpirationTimeInMS(node);
>         if (node.maintenanceNotExpired(maintenanceExpireTimeInMS)) {
>           datanodeAdminManager.startMaintenance(
>               node, maintenanceExpireTimeInMS);
>         } else if (hostConfigManager.isExcluded(node)) {
>           datanodeAdminManager.startDecommission(node);
>         } else {
>           datanodeAdminManager.stopMaintenance(node);
>           datanodeAdminManager.stopDecommission(node);
>         }
>       }
>       node.setUpgradeDomain(hostConfigManager.getUpgradeDomain(node));
>     }
>   }
> {code}
> All the isIncluded(), isExcluded() methods call node.getResolvedAddress() which does
the DNS lookup. We could probably change things to perform all the DNS lookups outside of
the write lock, and then take the lock and process the nodes. Also change or overload isIncluded()
etc to take the inetAddress rather than the datanode descriptor.
> It would not shorten the time the operation takes to run overall, but it would move the
long duration out of the write lock and avoid blocking the namenode for the entire time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message