flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prez Cannady <revp...@opencorrelate.org>
Subject Moving from single-node, Maven examples to cluster execution
Date Thu, 16 Jun 2016 21:51:18 GMT
Having a hard time trying to get my head around how to deploy my Flink programs to a pre-configured,
remote Flink cluster setup.

My Mavenized setup uses Spring Boot (to simplify class path handling and generate pretty logs)
to execute provision a StreamExecutionEnvironment with Kafka sources and sinks. I can also
run this quite effective the standard way (`java -jar …`).  What I’m unclear on is how
I might go about distributing this code to run on an existing Flink cluster setup.  Where
do I drop the jars? Do I need to restart Flink to do so?

class AppRunner extends CommandLineRunner {

    val log = LoggerFactory.getLogger(classOf[AppRunner])


    override def run(args: String*): Unit = {

        val env : StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment

    val consumer = …
    val producer = ...
        val stream = env.addSource(consumer)

        stream
            …
        // Do some stuff
        …
            .addSink(producer)

        env.execute
    }



}
…

@SpringBootApplication
object App {


    @throws(classOf[Exception])
    def main( args: Array[String] ) : Unit = {
        SpringApplication.run(classOf[AppRunner], args: _*)
    }
}


Try as I might, I couldn’t find any clear instructions on how to do this in the documentation.
 The cluster documentation ends with starting it.

https://ci.apache.org/projects/flink/flink-docs-release-0.8/cluster_setup.html#starting-flink
<https://ci.apache.org/projects/flink/flink-docs-release-0.8/cluster_setup.html#starting-flink>

The Wikiedits example doesn’t involve any third party dependencies, so I’m not clear on
how to manage class path for it.

https://ci.apache.org/projects/flink/flink-docs-release-1.0/quickstart/run_example_quickstart.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.0/quickstart/run_example_quickstart.html>

Any help in getting me on the right, preferably best practices path would be appreciated.


Prez Cannady  
p: 617 500 3378  
e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  










Mime
View raw message