flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Submitting jobs from within Scala code
Date Thu, 16 Jul 2015 10:27:01 GMT
Could you also look into the JobManager logs? You may be submitting a
corrupt JobGraph...

On Thu, Jul 16, 2015 at 11:45 AM, Till Rohrmann <trohrmann@apache.org>
wrote:

> When you run your program from the IDE, then you can specify a
> log4j.properties file. There you can configure where and what to log. It
> should be enough to place the log4j.properties file in the resource
> folder of your project. An example properties file could look like:
>
> log4j.rootLogger=INFO, testlogger
>
> log4j.appender.testlogger=org.apache.log4j.ConsoleAppender
> log4j.appender.testlogger.target = System.err
> log4j.appender.testlogger.layout=org.apache.log4j.PatternLayout
> log4j.appender.testlogger.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
>
> Alternatively, you can specify it via a JVM option: -Dlog4j.configuration=<path
> to properties file>
>
> Cheers,
> Till
> ​
>
> On Thu, Jul 16, 2015 at 11:23 AM, Philipp Goetze <
> philipp.goetze@tu-ilmenau.de> wrote:
>
>>  Hi Till,
>>
>> the problem is that this is the only output :( Or is it possible to get a
>> more verbose log output?
>>
>> Maybe it is important to note, that both Flink and our project is built
>> with Scala 2.11.
>>
>> Best Regards,
>> Philipp
>>
>>
>> On 16.07.2015 11:12, Till Rohrmann wrote:
>>
>> Hi Philipp,
>>
>>  could you post the complete log output. This might help to get to the
>> bottom of the problem.
>>
>>  Cheers,
>> Till
>>
>> On Thu, Jul 16, 2015 at 11:01 AM, Philipp Goetze <
>> philipp.goetze@tu-ilmenau.de> wrote:
>>
>>>  Hi community,
>>>
>>> in our project we try to submit built Flink programs to the jobmanager
>>> from within Scala code. The test program is executed correctly when
>>> submitted via the wrapper script "bin/flink run ..." and also with the
>>> webclient. But when executed from within the Scala code nothing seems to
>>> happen, but the following warning is found in the log:
>>>
>>> 10:47:18,153 WARN  akka.remote.ReliableDeliverySupervisor                   
    - Association with remote system [akka.tcp://flink@127.0.0.1:34074] has failed, address
is now gated for [5000] ms. Reason is: [org.apache.flink.runtime.jobgraph.AbstractJobVertex]
>>>
>>>
>>> Our submit method looks like that:
>>>
>>>  def submitJar(master: String, path: String, className: String, args: String*)
= {
>>>
>>>     val file = new File(path)
>>>
>>>     val parallelism = 1
>>>
>>>     val wait = true
>>>
>>>     try {
>>>
>>>       val program = new PackagedProgram(file, className, args:_*)
>>>
>>>       val jobManagerAddress = getInetFromHostport(master)
>>>
>>>       val client = new Client(jobManagerAddress, new Configuration(), program.getUserCodeClassLoader(),
1)
>>>
>>>       println("Executing " + path);
>>>
>>>       client.run(program, parallelism, wait);
>>>
>>>     } catch {
>>>
>>>       case e: ProgramInvocationException => e.printStackTrace()
>>>
>>>     }
>>>
>>>   }
>>>
>>>
>>> I took this as a reference:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-submit-flink-jars-from-plain-Java-programs-td656.html
>>>
>>> I hope you can help.
>>>
>>> Best Regards,
>>> Philipp Götze
>>>
>>
>>
>>
>

Mime
View raw message