spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jiangxb1987 <...@git.apache.org>
Subject [GitHub] spark pull request #22085: [SPARK-25095][PySpark] Python support for Barrier...
Date Mon, 13 Aug 2018 04:40:38 GMT
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22085#discussion_r209490553
  
    --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
    @@ -180,7 +183,42 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
             dataOut.writeInt(partitionIndex)
             // Python version of driver
             PythonRDD.writeUTF(pythonVer, dataOut)
    +        // Init a GatewayServer to port current BarrierTaskContext to Python side.
    +        val isBarrier = context.isInstanceOf[BarrierTaskContext]
    +        val secret = if (isBarrier) {
    +          Utils.createSecret(env.conf)
    +        } else {
    +          ""
    +        }
    +        val gatewayServer: Option[GatewayServer] = if (isBarrier) {
    +          Some(new GatewayServer.GatewayServerBuilder()
    +            .entryPoint(context.asInstanceOf[BarrierTaskContext])
    +            .authToken(secret)
    +            .javaPort(0)
    +            .callbackClient(GatewayServer.DEFAULT_PYTHON_PORT, GatewayServer.defaultAddress(),
    +              secret)
    +            .build())
    --- End diff --
    
    We have to port `BarrierTaskContext` from java to python side, otherwise there is no way
to call `BarrierTaskContext.barrier()` from python side. Thus, of course, the JavaGateway
is only initiated when the context is a `BarrierTaskContext`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message