flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Newport, Billy" <Billy.Newp...@gs.com>
Subject RE: Job Manager Configuration
Date Thu, 02 Nov 2017 12:38:23 GMT
The user code for all the flows is common though so is there an inefficiency here in terms
of copying this code for every operator?


From: Chesnay Schepler [mailto:chesnay@apache.org]
Sent: Wednesday, November 01, 2017 7:09 AM
To: user@flink.apache.org
Subject: Re: Job Manager Configuration

AFAIK there is no theoretical limit on the size of the plan, it just depends on the available
resources.

The job submissions times out since it takes too long to deploy all the operators that the
job defines. With 300 flows, each with 6 operators you're looking at potentially (1800 * parallelism)
tasks that have to be deployed. For each task Flink copies the user-code of all flows to the
executing TaskManager, which the network may just not be handle in time.

I suggest to split your job into smaller batches or even run each of them independently.

On 31.10.2017 16:25, Chan, Regina wrote:
Asking an additional question, what is the largest plan that the JobManager can handle? Is
there a limit? My flows don't need to run in parallel and can run independently. I wanted
them to run in one single job because it's part of one logical commit on my side.

Thanks,
Regina

From: Chan, Regina [Tech]
Sent: Monday, October 30, 2017 3:22 PM
To: 'user@flink.apache.org<mailto:user@flink.apache.org>'
Subject: Job Manager Configuration

Flink Users,

I have about 300 parallel flows in one job each with 2 inputs, 3 operators, and 1 sink which
makes for a large job. I keep getting the below timeout exception but I've already set it
to a 30 minute time out with a 6GB heap on the JobManager? Is there a heuristic to better
configure the job manager?

Caused by: org.apache.flink.runtime.client.JobClientActorSubmissionTimeoutException: Job submission
to the JobManager timed out. You may increase 'akka.client.timeout' in case the JobManager
needs more time to configure and confirm the job submission.

Regina Chan
Goldman Sachs - Enterprise Platforms, Data Architecture
30 Hudson Street, 37th floor | Jersey City, NY 07302 *  (212) 902-5697




Mime
View raw message