spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dieter (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23548) Redirect loop from Resourcemanager to Spark Webui
Date Thu, 01 Mar 2018 13:05:00 GMT
Dieter created SPARK-23548:
------------------------------

             Summary: Redirect loop from Resourcemanager to Spark Webui
                 Key: SPARK-23548
                 URL: https://issues.apache.org/jira/browse/SPARK-23548
             Project: Spark
          Issue Type: Bug
          Components: Web UI, YARN
    Affects Versions: 2.3.0
         Environment: *SW*

hadoop-3.0.0
Spark 2.3.0 (git revision a0d7949896) built for Hadoop 2.7.3

 

*Nodes*

jupyter
proxyserver
historyserver
resourcemanager
nodemanager ([http://9d581cfb391f:8042|http://9d581cfb391f:8042/])

 

*Config of sparkcontext*

spark.app.id=application_1519904229414_0011

spark.app.name=Spark shell

spark.driver.appUIAddress=http://jupyter:4040

spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties

spark.driver.host=jupyter

spark.driver.port=35813

spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties

spark.executor.id=driver

spark.home=/usr/local/spark

spark.jars=

spark.logConf=true

spark.master=yarn

spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS=proxyserver

spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES=[http://proxyserver:8089/proxy/application_1519904229414_0011] 

 

*yarn-site.xml*

<configuration>
 <!-- Site specific YARN configuration properties -->
 <property>
 <name>yarn.web-proxy.address</name>
 <value>proxyserver:8089</value>
 </property>
 <property>
 <name>yarn.resourcemanager.fs.state-store.uri</name>
 <value>/rmstate</value>
 </property>
 <property>
 <name>yarn.resourcemanager.webapp.address</name>
 <value>resourcemanager:8088</value>
 </property>
 <property>
 <name>yarn.timeline-service.generic-application-history.enabled</name>
 <value>true</value>
 </property>
 <property>
 <name>yarn.resourcemanager.recovery.enabled</name>
 <value>true</value>
 </property>
 <property>
 <name>yarn.nodemanager.bind-host</name>
 <value>0.0.0.0</value>
 </property>
 <property>
 <name>yarn.timeline-service.bind-host</name>
 <value>0.0.0.0</value>
 </property>
 <property>
 <name>yarn.log-aggregation-enable</name>
 <value>true</value>
 </property>
 <property>
 <name>yarn.timeline-service.enabled</name>
 <value>true</value>
 </property>
 <property>
 <name>yarn.resourcemanager.store.class</name>
 <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
 </property>
 <property>
 <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
 <value>true</value>
 </property>
 <property>
 <name>yarn.nodemanager.remote-app-log-dir</name>
 <value>/app-logs</value>
 </property>
 <property>
 <name>yarn.nodemanager.vmem-pmem-ratio</name>
 <value>4.1</value>
 </property>
 <property>
 <name>yarn.resourcemanager.resource-tracker.address</name>
 <value>resourcemanager:8031</value>
 </property>
 <property>
 <name>yarn.resourcemanager.hostname</name>
 <value>resourcemanager</value>
 </property>
 <property>
 <name>yarn.timeline-service.hostname</name>
 <value>historyserver</value>
 </property>
 <property>
 <name>yarn.log.server.url</name>
 <value>http://historyserver:8188/applicationhistory/logs/</value>
 </property>
 <property>
 <name>yarn.resourcemanager.bind-host</name>
 <value>0.0.0.0</value>
 </property>
 <property>
 <name>yarn.resourcemanager.scheduler.address</name>
 <value>resourcemanager:8030</value>
 </property>
 <property>
 <name>yarn.resourcemanager.address</name>
 <value>resourcemanager:8032</value>
 </property>
</configuration>
            Reporter: Dieter


Access from Resourcemanager to Webui in a distributed environment results in a redirect loop.
It seems webui is not recognizing request is coming from resourcemanager proxy.

 

*Situation*

Spark application running on jupyter

 

*link in resourcemanger:8088 (behind Trackingui->Applicationmaster)*

[http://proxyserver:8089/proxy/application_1519904229414_0011/]

 

*Response of webui on jupyter:4040 (redirects to proxyserver)*

{{<html>}}
{{  <head>}}
{{    <title>      Moved    </title>  </head>  <body>   
<h1>      Moved    </h1>    <div>      Content has moved     
<a href="http://proxyserver:8089/proxy/application_1519904229414_0011/">here</a>   
</div>  }}
{{  </body>}}
{{</html>}}{{ }}

 

*log output of proxyserver (results in redirect)*

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,905 DEBUG strategy.ExecuteProduceConsume: EPC Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a
producing     

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG protocol.RequestAuthCache: Auth cache not
set in the context            

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.wire: >> "[\r][\n]"            

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> GET / HTTP/1.1          

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8     

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> Host: jupyter:4040  

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.headers: << Server: Jetty(9.3.z-SNAPSHOT)          

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG client.DefaultRedirectStrategy: Redirect
requested to location 'http://proxyserver:8089/proxy/application_1519904229414_0001/'     

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " Moved[\n]"    

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " Content has moved
[\n]"          

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " <a href="http://proxyserver:8089/proxy/application_1519904229414_0001/">here</a>[\n]"        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG conn.BasicClientConnectionManager: Connection
can be kept alive indefinitely           

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG conn.DefaultClientConnectionOperator: Connecting
to proxyserver:8089        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Queued change org.eclipse.jetty.io.ManagedSelector$Accept@5498de79
on org.eclipse.jetty.io.ManagedSelector@558bdf1f id=0 keys=63 selected=0       

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Selector loop woken
up from select, 0/63 selected          

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,913 DEBUG strategy.ExecuteProduceConsume: EPC Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a
dispatch        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,913 DEBUG thread.QueuedThreadPool: queue EPC Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a          

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,913 DEBUG io.SelectChannelEndPoint: changeInterests
p=false 0->1 for SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,FI,-,0/10000,HttpConnection@7d134d7f}{io=0/1,kio=0,kro=0}        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG conn.DefaultClientConnection: Sending request:
GET /proxy/application_1519904229414_0001/ HTTP/1.1   

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Accept-Encoding: gzip,
deflate[\r][\n]"

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Host: proxyserver:8089[\r][\n]"             

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG http.headers: >> GET /proxy/application_1519904229414_0001/
HTTP/1.1    

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG io.FillInterest: FillInterest@301a7eba\{true,AC.ReadCB@7d134d7f{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}}
fillable AC.ReadCB@7d134d7f\{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}      

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG io.ChannelEndPoint: filled 426 SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG server.HttpConnection: HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,0/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0
of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]
filled 426 HeapByteBuffer@5b3c5d02[p=0,l=426,c=8192,r=426]=\{<<<GET /proxy/applic...
Keep-Alive\r\n\r\n>>>Accept-Language: ...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}          

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: SPACE2 --> REQUEST_VERSION         

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE -->
HEADER        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE -->
HEADER        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER --> HEADER_VALUE

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG server.HttpConnection: HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7\{/10.0.1.17:48699<->8089,Open,in,out,-,-,0/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=END,0
of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=1,c=false,a=IDLE,uri=//proxyserver:8089/proxy/application_1519904229414_0001/}]
parsed true HttpParser\{s=END,0 of 0}   

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,916 DEBUG server.session: session=null   

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,916 DEBUG servlet.ServletHandler: call filter static_user_filter     

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,918 DEBUG io.ManagedSelector: Selector loop waiting
on select  

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,929 DEBUG webproxy.WebAppProxyServlet: local InetAddress
for proxy host: proxyserver/10.0.1.17  

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,930 DEBUG conn.BasicClientConnectionManager: Get
connection for route proxyserver/10.0.1.17->{}->http://jupyter:4040

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,933 DEBUG protocol.RequestTargetAuthentication: Target
auth state: UNCHALLENGED   

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG conn.DefaultClientConnection: Sending request:
GET / HTTP/1.1          

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8[\r][\n]"     

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "User-Agent: Mozilla/5.0
(Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167
Safari/537.36[\r][\n]"             

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Host: jupyter:4040[\r][\n]"      

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"             

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.headers: >> Connection: Keep-Alive        

March 1st 2018, 12:38:50.000     /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl            
proxyserver stderr | 2018-03-01 11:38:49,937 DEBUG http.headers: << Location: http://proxyserver:8089/proxy/application_1519904229414_0001/       

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message