flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: how many 'run -c' commands to start?
Date Sat, 30 Sep 2017 21:58:33 GMT
The order in which you pass program arguments is quite important. Can 
you try again putting --detached directly after run?
I.e. flink run --detached -c ...

The reason being that arguments after the jar are treated as arguments 
for the user-program, and not as arguments for the job execution.

On 29.09.2017 17:23, r. r. wrote:
> Sure, here is the cmdline output:
>
> flink-1.3.2/bin/flink run -c com.corp.flink.KafkaJob quickstart.jar --topic InputQ --bootstrap.servers
localhost:9092 --zookeeper.connect localhost:2181 --group.id Consumers -p 5 --detached
> Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
> Using address localhost:6123 to connect to JobManager.
> JobManager web interface address http://localhost:8081
> Starting execution of program
> Submitting job with JobID: 5cc74547361cec2ff9874764dac9ee91. Waiting for job completion.
> Connected to JobManager at Actor[akka.tcp://flink@localhost:6123/user/jobmanager#-100905913]
with leader session id 00000000-0000-0000-0000-000000000000.
> 09/28/2017 16:30:40    Job execution switched to status RUNNING.
> 09/28/2017 16:30:40    Source: Custom Source -> Map(1/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Source: Custom Source -> Map(2/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Source: Custom Source -> Map(3/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Source: Custom Source -> Map(4/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Source: Custom Source -> Map(5/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Sink: Unnamed(1/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Sink: Unnamed(2/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Sink: Unnamed(3/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Sink: Unnamed(4/5) switched to SCHEDULED
> 09/28/2017 16:30:40    Sink: Unnamed(5/5) switched to SCHEDULED
> ...
>
> I thought --detach will put the process in the background, and give me back the cmdline,
but maybe I got the meaning behind this option wrong?
>
> Thank you!
>
>
>
>
>
>
>
>   >-------- Оригинално писмо --------
>
>   >От: Chesnay Schepler chesnay@apache.org
>
>   >Относно: Re: how many 'run -c' commands to start?
>
>   >До: user@flink.apache.org
>
>   >Изпратено на: 29.09.2017 18:01
>
>
>
>   
>   
>>     
>   
>>   
>   
>>    
>   
>>    
>   
>>     The only nodes that matter are those on which the Flink processes,  i.e
>   
>>     Job- and TaskManagers, are being run.
>   
>>      
>   
>>      To prevent a JobManager node failure from causing the job to fail you have to
look into an
>   
>>     HA setup.
>   
>>      (The jobmanager is responsible for distributing/coordinating work)
>   
>>      
>   
>>      In case of a TaskManager node failure the job will fail and
>   
>>     restart, provided that enough TaskManagers are
>   
>>      still alive to satisfy the resource requirements of the job.
>   
>>      
>   
>>      
>   
>>      Can you elaborate a bit more what happened when you used the  --detached param?
>   
>>      
>   
>>      On 28.09.2017 16:33, r. r. wrote:
>   
>>      
>   
>>     
>   
>>     
>   
>>     Thank you, Chesnay
>   
>> to make sure - should the node where the job has been submitted goes down, the processing
will continue, I hope?
>   
>> Do I need to ensure this by configuration?
>   
>   
>> btw I added --detached param to the run cmd, but it didn't go into background process
as I would've expected. Am I guessing wrong?
>   
>   
>> Thanks!
>   
>> Rob
>   
>   
>>   >-------- Оригинално писмо --------
>   
>   
>>   >От: Chesnay Schepler chesnay@apache.org
>   
>   
>>   >Относно: Re: how many 'run -c' commands to start?
>   
>   
>>   >До: user@flink.apache.org
>   
>   
>>   >Изпратено на: 28.09.2017 15:05
>   
>   
>>   
>   
>>   
>   
>>      
>   
>>      Hi!
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>      Given a Flink cluster, you would only call `flink run ...` to submit a
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>      job once; for simplicity i would submit it on the node where you started
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>      the cluster. Flink will automatically distribute job across the cluster,
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>      in smaller independent parts known as Tasks.
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>      Regards,
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>      Chesnay
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>      On 28.09.2017 08:31, r. r. wrote:
>   
>>   
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       Hello
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       I successfully ran a job with 'flink run -c', but this is for the local
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       setup.
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       How should i proceed with a cluster? Will flink automagically instantiate
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       the job on all servers - i hope i don't have to start 'flink run -c' on all
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       machines.
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       New to flink and bigdata, so sorry for the probably silly question
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       Thanks!
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>       Rob
>   
>>   
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>   
>   
>>      
>   
>>       
>   
>>        
>   
>>       
>   
>>      
>   
>>      
>   
>>     
>   
>>        
>   
>>   



Mime
View raw message