flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "r. r." <rob...@abv.bg>
Subject Re: how many 'run -c' commands to start?
Date Mon, 02 Oct 2017 12:10:35 GMT
Thanks, Chesnay, that was indeed the problem.
It also explains why -p5 was not working for me from the cmdline

Best regards
Robert






 >-------- Оригинално писмо --------

 >От: Chesnay Schepler chesnay@apache.org

 >Относно: Re: how many 'run -c' commands to start?

 >До: "r. r." <robert@abv.bg>

 >Изпратено на: 01.10.2017 00:58



 
> The order in which you pass program arguments is quite important. Can 
 
> you try again putting --detached directly after run?
 
> I.e. flink run --detached -c ...
 
> 
 
> The reason being that arguments after the jar are treated as arguments 
 
> for the user-program, and not as arguments for the job execution.
 
> 
 
> On 29.09.2017 17:23, r. r. wrote:
 
> > Sure, here is the cmdline output:
 
> >
 
> > flink-1.3.2/bin/flink run -c com.corp.flink.KafkaJob quickstart.jar --topic InputQ
--bootstrap.servers localhost:9092 --zookeeper.connect localhost:2181 --group.id Consumers
-p 5 --detached
 
> > Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
 
> > Using address localhost:6123 to connect to JobManager.
 
> > JobManager web interface address http://localhost:8081
 
> > Starting execution of program
 
> > Submitting job with JobID: 5cc74547361cec2ff9874764dac9ee91. Waiting for job completion.
 
> > Connected to JobManager at Actor[akka.tcp://flink@localhost:6123/user/jobmanager#-100905913]
with leader session id 00000000-0000-0000-0000-000000000000.
 
> > 09/28/2017 16:30:40    Job execution switched to status RUNNING.
 
> > 09/28/2017 16:30:40    Source: Custom Source -> Map(1/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Source: Custom Source -> Map(2/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Source: Custom Source -> Map(3/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Source: Custom Source -> Map(4/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Source: Custom Source -> Map(5/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Sink: Unnamed(1/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Sink: Unnamed(2/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Sink: Unnamed(3/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Sink: Unnamed(4/5) switched to SCHEDULED
 
> > 09/28/2017 16:30:40    Sink: Unnamed(5/5) switched to SCHEDULED
 
> > ...
 
> >
 
> > I thought --detach will put the process in the background, and give me back the
cmdline, but maybe I got the meaning behind this option wrong?
 
> >
 
> > Thank you!
 
> >
 
> >
 
> >
 
> >
 
> >
 
> >
 
> >
 
> >   >-------- Оригинално писмо --------
 
> >
 
> >   >От: Chesnay Schepler chesnay@apache.org
 
> >
 
> >   >Относно: Re: how many 'run -c' commands to start?
 
> >
 
> >   >До: user@flink.apache.org
 
> >
 
> >   >Изпратено на: 29.09.2017 18:01
 
> >
 
> >
 
> >
 
> >   
 
> >   
 
> >>     
 
> >   
 
> >>   
 
> >   
 
> >>    
 
> >   
 
> >>    
 
> >   
 
> >>     The only nodes that matter are those on which the Flink processes,  i.e
 
> >   
 
> >>     Job- and TaskManagers, are being run.
 
> >   
 
> >>      
 
> >   
 
> >>      To prevent a JobManager node failure from causing the job to fail you have
to look into an
 
> >   
 
> >>     HA setup.
 
> >   
 
> >>      (The jobmanager is responsible for distributing/coordinating work)
 
> >   
 
> >>      
 
> >   
 
> >>      In case of a TaskManager node failure the job will fail and
 
> >   
 
> >>     restart, provided that enough TaskManagers are
 
> >   
 
> >>      still alive to satisfy the resource requirements of the job.
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>      Can you elaborate a bit more what happened when you used the  --detached
param?
 
> >   
 
> >>      
 
> >   
 
> >>      On 28.09.2017 16:33, r. r. wrote:
 
> >   
 
> >>      
 
> >   
 
> >>     
 
> >   
 
> >>     
 
> >   
 
> >>     Thank you, Chesnay
 
> >   
 
> >> to make sure - should the node where the job has been submitted goes down, the
processing will continue, I hope?
 
> >   
 
> >> Do I need to ensure this by configuration?
 
> >   
 
> >   
 
> >> btw I added --detached param to the run cmd, but it didn't go into background
process as I would've expected. Am I guessing wrong?
 
> >   
 
> >   
 
> >> Thanks!
 
> >   
 
> >> Rob
 
> >   
 
> >   
 
> >>   >-------- Оригинално писмо --------
 
> >   
 
> >   
 
> >>   >От: Chesnay Schepler chesnay@apache.org
 
> >   
 
> >   
 
> >>   >Относно: Re: how many 'run -c' commands to start?
 
> >   
 
> >   
 
> >>   >До: user@flink.apache.org
 
> >   
 
> >   
 
> >>   >Изпратено на: 28.09.2017 15:05
 
> >   
 
> >   
 
> >>   
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      Hi!
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      Given a Flink cluster, you would only call `flink run ...` to submit a
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      job once; for simplicity i would submit it on the node where you started
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      the cluster. Flink will automatically distribute job across the cluster,
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      in smaller independent parts known as Tasks.
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      Regards,
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      Chesnay
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      On 28.09.2017 08:31, r. r. wrote:
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       Hello
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       I successfully ran a job with 'flink run -c', but this is for the local
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       setup.
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       How should i proceed with a cluster? Will flink automagically instantiate
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       the job on all servers - i hope i don't have to start 'flink run -c' on
all
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       machines.
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       New to flink and bigdata, so sorry for the probably silly question
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       Thanks!
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>       Rob
 
> >   
 
> >>   
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>   
 
> >   
 
> >>      
 
> >   
 
> >>       
 
> >   
 
> >>        
 
> >   
 
> >>       
 
> >   
 
> >>      
 
> >   
 
> >>      
 
> >   
 
> >>     
 
> >   
 
> >>        
 
> >   
 
> >>   

Mime
View raw message