beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Work logged] (BEAM-313) Enable the use of an existing spark context with the SparkPipelineRunner
Date Mon, 27 Aug 2018 11:40:00 GMT


ASF GitHub Bot logged work on BEAM-313:

                Author: ASF GitHub Bot
            Created on: 27/Aug/18 11:39
            Start Date: 27/Aug/18 11:39
    Worklog Time Spent: 10m 
      Work Description: amarouni commented on issue #401: [BEAM-313] Enable the use of an
existing spark context with the SparkPipelineRunner
   Here's an old Scala snippet that shows how to use Beam & SJS, it's based on old versions
of SJS & Beam so it probably won't compile with new SJS/Beam versions but you'll get the
idea : 
   import com.typesafe.config.Config
   import org.apache.beam.runners.spark.{ SparkContextOptions, SparkRunner }
   import org.apache.beam.sdk.Pipeline
   import org.apache.beam.sdk.coders.StringUtf8Coder
   import org.apache.beam.sdk.options.PipelineOptionsFactory
   import org.apache.beam.sdk.transforms.Create
   import org.apache.spark.SparkContext
   import spark.jobserver.{ SparkJob, SparkJobInvalid, SparkJobValid, SparkJobValidation }
   import scala.collection.JavaConversions
   import scala.util.Try
    * Beam wordcount test. Returns the word count of a fixed String seq.
   object BeamWordCount extends SparkJob {
     override def validate(sc: SparkContext, config: Config): SparkJobValidation = {
         .map(x => SparkJobValid)
         .getOrElse(SparkJobInvalid("No wordList in context config"))
     override def runJob(sc: SparkContext, jobConfig: Config): Any = {
       // Input test list
       val inputBuffer = scala.collection.JavaConversions.asScalaBuffer(jobConfig.getStringList("wordList"))
       val WORDS = inputBuffer.toList
       // Pipeline options
       val sparkPipelineOptions =[SparkContextOptions])
       sparkPipelineOptions.setAppName("Beam WordCount test")
       sparkPipelineOptions.setProvidedSparkContext(new JavaSparkContext(sc))
       // Pipeline
       val pipeline = Pipeline.create(sparkPipelineOptions)
       // Input + processing + Output
       val output = pipeline
         .apply(new CountWords())
       // Result
       // val result: EvaluationResult =[EvaluationResult]
       // Run job & wait until finish
   It'd be nice to contribute this as new documentation if you manage to get it to work. 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 138351)
    Time Spent: 1h 10m  (was: 1h)

> Enable the use of an existing spark context with the SparkPipelineRunner
> ------------------------------------------------------------------------
>                 Key: BEAM-313
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-spark
>            Reporter: Abbass Marouni
>            Assignee: Jean-Baptiste Onofré
>            Priority: Major
>             Fix For: 0.3.0-incubating
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
> The general use case is that the SparkPipelineRunner creates its own Spark context and
uses it for the pipeline execution.
> Another alternative is to provide the SparkPipelineRunner with an existing spark context.
This can be interesting for a lot of use cases where the Spark context is managed outside
of beam (context reuse, advanced context management, spark job server, ...).
> code sample :

This message was sent by Atlassian JIRA

View raw message