Mailing-List: contact dev-help@gearpump.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@gearpump.incubator.apache.org
Date: Fri, 20 Oct 2017 16:05:00 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: dev@gearpump.incubator.apache.org
Message-ID: <JIRA.13108540.1507713863000.19101.1508515500753@Atlassian.JIRA>
In-Reply-To: <JIRA.13108540.1507713863000@Atlassian.JIRA>
References: <JIRA.13108540.1507713863000@Atlassian.JIRA> <JIRA.13108540.1507713863435@jira-lw-us.apache.org>
Subject: [jira] [Commented] (GEARPUMP-355) AppMasterResolver fails to run
 against a kerberized Hadoop cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 20 Oct 2017 16:05:09 -0000


    [ https://issues.apache.org/jira/browse/GEARPUMP-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212823#comment-16212823 ] 

ASF GitHub Bot commented on GEARPUMP-355:
-----------------------------------------

Github user titikakatoo commented on a diff in the pull request:

    https://github.com/apache/incubator-gearpump/pull/231#discussion_r146003668
  
    --- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
    @@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
         result
       }
     }
    +
    +object AppMasterResolver {
    +  val LOG = LogUtil.getLogger(getClass)
    +
    +  def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
    +    val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
    +    LOG.info(s"appMasterPath=$appMasterPath")
    +
    +    val connectionFactory: URLConnectionFactory = URLConnectionFactory
    +      .newDefaultURLConnectionFactory(new YarnConfiguration())
    +    val url: URL = new URL(appMasterPath)
    +    val connection: HttpURLConnection = connectionFactory.openConnection(url)
    +      .asInstanceOf[HttpURLConnection]
    +    connection.setInstanceFollowRedirects(true)
    +
    +    try {
    +      connection.connect()
    +    } catch {
    +      case e: IOException =>
    +        LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
    +    }
    +
    +    val status = connection.getResponseCode
    +    if (status == 200) {
    +      val stream: java.io.InputStream = connection.getInputStream
    +      val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
    +      LOG.info("Successfully resolved AppMaster address: " + response)
    +      connection.disconnect()
    +      AkkaHelper.actorFor(system, response)
    +    } else {
    +      connection.disconnect()
    +      throw new IOException("Fail to resolve AppMaster address, please make sure " +
    +        s"${report.getOriginalTrackingUrl} is accessible...")
    --- End diff --
    
    @manuzhang getTrackingURL was merged already in the master branch.


> AppMasterResolver fails to run against a kerberized Hadoop cluster
> ------------------------------------------------------------------
>
>                 Key: GEARPUMP-355
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-355
>             Project: Apache Gearpump
>          Issue Type: Bug
>          Components: security, yarn
>    Affects Versions: 0.8.4
>            Reporter: Timea Magyar
>            Assignee: Timea Magyar
>             Fix For: 0.8.4
>
>
> When trying to launch a Gearpump cluster in a kerberized Hadoop/Yarn environment, after the Application Master address has been resolved as a prerequisite, the YarnAppMaster (responsible for starting GearPump masters, workers, UI servers as Yarn containers) address (actor reference) must be obtained via Kerberos/Spnego. (Kerberos over http)
> The current implementation for this resides in the AppMasterResolver class and is using an apache http client (version 3.x) for establishing a connection to the Application Master and obtain the above YarnAppMaster actor reference. Since the apache http client does not support the negotiate authentication scheme in version 3.x (required for a connection over kerberos/spnego) this step will always fail in a kerberized Yarn/Hadoop cluster set-up.
> I tested this in a secured/kerberized CDH 5.7.5 environment.  I would like to provide a patch for this  by adapting the SPNEGO-enabled Hadoop web connection code from WebHDFS.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)