gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run against a kerberized Hadoop cluster
Date Thu, 12 Oct 2017 09:48:01 GMT

    [ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201722#comment-16201722
] 

ASF GitHub Bot commented on GEARPUMP-355:
-----------------------------------------

Github user titikakatoo commented on the issue:

    https://github.com/apache/incubator-gearpump/pull/231
  
    @huafengw  
    I can provide you the following sanitized extract from the logs when trying to use gearpump
on a kerberized hadoop cluster without the patch:
    
    
    > [HttpClient] Java version: 1.8.0_141
    > [HttpClient] Java vendor: Oracle Corporation
    > [HttpClient] Java class path: <classpath>
    > [HttpClient] Operating system name: <>
    > [HttpClient] Operating system architecture: <>
    > [HttpClient] Operating system version: <>
    > [HttpClient] SUN 1.8: <>
    > [HttpClient] SunRsaSign 1.8: <>
    > [HttpClient] SunEC 1.8: <>
    > [HttpClient] SunJSSE 1.8: <>
    > [HttpClient] SunJCE 1.8: <>
    > [HttpClient] SunJGSS 1.8: Sun **(Kerberos v5, SPNEGO)**
    > [HttpClient] SunSASL 1.8: <>
    > [HttpClient] XMLDSig 1.8: <>
    > [HttpClient] SunPCSC 1.8: <>
    > [HttpClient] SaslPlainServer 1.0: SASL PLAIN Authentication Server
    > [DefaultHttpParams] Set parameter http.useragent = **Jakarta Commons-HttpClient/3.1**
    > [DefaultHttpParams] Set parameter http.protocol.version = HTTP/1.1
    > [DefaultHttpParams] Set parameter http.connection-manager.class = class org.apache.commons.httpclient.SimpleHttpConnectionManager
    > [DefaultHttpParams] Set parameter http.protocol.cookie-policy = default
    > [DefaultHttpParams] Set parameter http.protocol.element-charset = US-ASCII
    > [DefaultHttpParams] Set parameter http.protocol.content-charset = ISO-8859-1
    > [DefaultHttpParams] Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@54f5f647
    > [DefaultHttpParams] Set parameter http.dateparser.patterns = <>
    > [AppMasterResolver] appMasterPath=https://<proxy_host>:<proxy_port>/proxy/<application_id>//supervisor-actor-path
    > [HttpConnection] Open connection to <proxy_host>:<proxy_port>
    > [header] >> "GET /proxy/<application_id>//supervisor-actor-path HTTP/1.1[\r][\n]"
    > [HttpMethodBase] Adding Host request header
    > [header] >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
    > [header] >> "Host: <proxy_host>:<proxy_port>[\r][\n]"
    > [header] >> "[\r][\n]"
    > [header] << "HTTP/1.1 401 Authentication required[\r][\n]"
    > [header] << "HTTP/1.1 401 Authentication required[\r][\n]"
    > [header] << "Cache-Control: must-revalidate,no-cache,no-store[\r][\n]"
    > [header] << "Date: Thu, 12 Oct 2017 09:06:05 GMT[\r][\n]"
    > [header] << "Pragma: no-cache[\r][\n]"
    > [header] << "Date: Thu, 12 Oct 2017 09:06:05 GMT[\r][\n]"
    > [header] << "Pragma: no-cache[\r][\n]"
    > [header] << "Content-Type: text/html; charset=iso-8859-1[\r][\n]"
    > [header] << "X-FRAME-OPTIONS: SAMEORIGIN[\r][\n]"
    > [header] << "WWW-Authenticate: Negotiate[\r][\n]"
    > [header] << "Set-Cookie: hadoop.auth=; Path=/; Secure; HttpOnly[\r][\n]"
    > [header] << "Content-Length: 1452[\r][\n]"
    > [header] << "Server: Jetty(6.1.26.cloudera.4)[\r][\n]"
    > [header] << "[\r][\n]"
    > [CookieSpec] Unrecognized cookie attribute: name=HttpOnly, value=null
    > [HttpMethodBase] Cookie accepted: "$Version=0; hadoop.auth=; $Path=/"
    > [HttpMethodDirector] Authorization required
    > [AuthChallengeProcessor] Supported authentication schemes in the order of preference:
**[ntlm, digest, basic]**
    > [AuthChallengeProcessor] Challenge for ntlm authentication scheme not available
    > [AuthChallengeProcessor] Challenge for digest authentication scheme not available
    > [AuthChallengeProcessor] Challenge for basic authentication scheme not available
    > [HttpMethodDirector] **Unable to respond to any of these challenges: {negotiate=Negotiate}**
    > [AppMasterResolver] Failed to connect YarnAppMaster(tried 1)... Fail to resolve AppMaster
address, please make sure https://<proxy_host>:<proxy_port>/proxy/<application_id>/
is accessible...
    
    
    Currently gearpump uses the apache http client version 3.1. This client only supports
the authentication schemes [ntlm, digest, basic] (see logs). As you can further see in the
logs: Unable to respond to any of these challenges: {negotiate=Negotiate}, a communication
via kerberos spnego is not possible in this case.
    
    After the patch trying to get the active configuration from YarnAppmaster produces the
following sanitized logs:
    
    > [Client] IPC Client (<>) connection to <host><port> from kerberos
principal: starting, having connections 1
    > [SaslRpcClient] reading next wrapped RPC packet
    > [Client] IPC Client (<>) connection to <host><port> from kerberos
principal sending #0
    > [SaslRpcClient] wrapping token of length:<length>
    > [SaslRpcClient] unwrapping token of length:<length>
    > [Client] IPC Client (<>) connection to <host><port> from kerberos
principal got value #0
    > [ProtobufRpcEngine] Call: getApplicationReport took 224ms
    > [AppMasterResolver$] appMasterPath=https://<host>:<port>/proxy/application_<id>/supervisor-actor-path
    > [FileBasedKeyStoresFactory] CLIENT TrustStore: <client_trust_store>
    > [ReloadingX509TrustManager] Loaded truststore '<client_trust_store>'
    > [FileBasedKeyStoresFactory] CLIENT Loaded TrustStore: <client_trust_store>
    > [URLConnectionFactory] open URL connection
    > [AppMasterResolver$] **Successfully resolved AppMaster address: akka.tcp://GearpumpAM@<host>:<port>/user/appMaster**
    > **ActiveConfig(Config<config>)**
    > [RemoteActorRefProvider$RemotingTerminator] Shutting down remote daemon.
    > [RemoteActorRefProvider$RemotingTerminator] Remote daemon shut down; proceeding with
flushing remote transports.
    > [RemoteActorRefProvider$RemotingTerminator] Remoting shut down.
    
    
    
    
    
    
    
    



> AppMasterResolver fails to run against a kerberized Hadoop cluster
> ------------------------------------------------------------------
>
>                 Key: GEARPUMP-355
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-355
>             Project: Apache Gearpump
>          Issue Type: Bug
>          Components: security, yarn
>    Affects Versions: 0.8.4
>            Reporter: Timea Magyar
>             Fix For: 0.8.4
>
>
> When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn environment, after
the Application Master address has been resolved as a prerequisite, the YarnAppMaster (responsible
for starting GearPump masters, workers, UI servers as Yarn containers) address (actor reference)
must be obtained via Kerberos/Spnego. (Kerberos over http)
> The current implementation for this resides in the AppMasterResolver class and is using
an apache http client (version 3.x) for establishing a connection to the Application Master
and obtain the above YarnAppMaster actor reference. Since the apache http client does not
support the negotiate authentication scheme in version 3.x (required for a connection over
kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop cluster set-up.
> I tested this in a secured/kerberized CDH 5.7.5 environment.  I would like to provide
a patch for this  by adapting the SPNEGO-enabled Hadoop web connection code from WebHDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message