spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Darabos (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup
Date Thu, 03 Mar 2016 13:07:18 GMT

    [ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177784#comment-15177784
] 

Daniel Darabos commented on SPARK-13620:
----------------------------------------

I just tested with the latest 2.0 nightly. Starting {{spark-shell}} is subjectively fast.
{{strace}} does not see the {{RESOLVE-ADDRESS}} request. Looks like it's fixed then! I should
have checked {{master}} before filing an issue, sorry.

> Avoid reverse DNS lookup for 0.0.0.0 on startup
> -----------------------------------------------
>
>                 Key: SPARK-13620
>                 URL: https://issues.apache.org/jira/browse/SPARK-13620
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>    Affects Versions: 1.6.0
>            Reporter: Daniel Darabos
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> I noticed we spend 5+ seconds during application startup with the following stack trace:
> {code}
>         at java.net.Inet6AddressImpl.getHostByAddr(Native Method)
>         at java.net.InetAddress$1.getHostByAddr(InetAddress.java:926)
>         at java.net.InetAddress.getHostFromNameService(InetAddress.java:611)
>         at java.net.InetAddress.getHostName(InetAddress.java:553)
>         at java.net.InetAddress.getHostName(InetAddress.java:525)
>         at java.net.InetSocketAddress$InetSocketAddressHolder.getHostName(InetSocketAddress.java:82)
>         at java.net.InetSocketAddress$InetSocketAddressHolder.access$600(InetSocketAddress.java:56)
>         at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:345)
>         at org.spark-project.jetty.server.Server.<init>(Server.java:115)
>         at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:243)
>         at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
>         at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
>         at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1964)
>         at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
>         at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1955)
>         at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)
>         at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
>         at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
>         at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
>         at scala.Option.foreach(Option.scala:236)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:481)
> {code}
> Spark wants to start a server on localhost. So it [creates an {{InetSocketAddress}}|https://github.com/apache/spark/blob/v1.6.0/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L243]
[with hostname {{"0.0.0.0"}}|https://github.com/apache/spark/blob/v1.6.0/core/src/main/scala/org/apache/spark/ui/WebUI.scala#L136].
Spark passes in a hostname string, but Java [recognizes that it's actually an address|https://github.com/openjdk-mirror/jdk/blob/adea42765ae4e7117c3f0e2d618d5e6aed44ced2/src/share/classes/java/net/InetSocketAddress.java#L220]
and so sets the hostname to {{null}}. So when Jetty [calls {{getHostName}}|https://github.com/eclipse/jetty.project/blob/jetty-8.1.14.v20131031/jetty-server/src/main/java/org/eclipse/jetty/server/Server.java#L115]
Java has to do a reverse DNS lookup for {{0.0.0.0}}. That takes 5+ seconds on my machine.
Maybe it's just me? It's a very vanilla Ubuntu 14.04.
> There is a simple fix. Instead of passing in {{"0.0.0.0"}} we should not set a hostname.
In this case [{{InetAddress.anyLocalAddress()}} is used|https://github.com/openjdk-mirror/jdk/blob/adea42765ae4e7117c3f0e2d618d5e6aed44ced2/src/share/classes/java/net/InetSocketAddress.java#L166],
which is the same, but does not need resolving.
> {code}
> scala> { val t0 = System.currentTimeMillis; new java.net.InetSocketAddress("0.0.0.0",
8000).getHostName; System.currentTimeMillis - t0 }
> res0: Long = 5432
> scala> { val t0 = System.currentTimeMillis; new java.net.InetSocketAddress(8000).getHostName;
System.currentTimeMillis - t0 }
> res1: Long = 0
> {code}
> I'll send a pull request for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message