flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek R. Singh" <abhis...@tetrationanalytics.com>
Subject Re: weird client failure/timeout
Date Mon, 23 Jan 2017 16:20:20 GMT
Is there a limit on how many DataStreams can be defined in a streaming program?

Looks like flink has problems handling too many data streams? I simplified my topology further.
For eg, this works (parallelism of 4)



However, when I try to go beyond 51 (found empirically by parametrizing nParts), it barfs
again. Submission fails, it wants me to increase akka.client.timeout

Here is the reduced code for repro (union at the end itself is not an issue). It is the parallelism
of the first for loop:
int nParts = cfg.getInt("dummyPartitions", 4);
boolean combineAtEnd = cfg.getBoolean("dummyCombineAtEnd", true);

// create lots of streams
List<SingleOutputStreamOperator<String>> streams = new ArrayList<>(nParts);
for (int i = 0; i < nParts; i++) {
  streams.add(env
      .readFile(
          new TextInputFormat(new Path("/tmp/input")),
          "/tmp/input",
          FileProcessingMode.PROCESS_CONTINUOUSLY,
          1000,
          FilePathFilter.createDefaultFilter())
      .setParallelism(1).name("src"));
}

if (combineAtEnd == true) {
  DataStream<String> combined = streams.get(0);
  for (int i = 1; i < nParts; i++) {
    combined = combined.union(streams.get(i));
  }
  combined.print().setParallelism(1);
} else { // die parallel
  for (int i = 1; i < nParts; i++) {
    streams.get(i).print();
  }
}


> On Jan 23, 2017, at 6:14 AM, Abhishek R. Singh <abhishsi@tetrationanalytics.com>
wrote:
> 
> I even make it 10 minutes:
> 
> akka.client.timeout: 600s
> 
> But doesn’t feel like it is taking effect. It still comes out at about the same time
with the same error.
> 
> -Abhishek-
> 
>> On Jan 23, 2017, at 6:04 AM, Abhishek R. Singh <abhishsi@tetrationanalytics.com
<mailto:abhishsi@tetrationanalytics.com>> wrote:
>> 
>> yes, I had increased it to 5 minutes. It just sits there and bails out again.
>> 
>>> On Jan 23, 2017, at 1:47 AM, Jonas <jonas@huntun.de <mailto:jonas@huntun.de>>
wrote:
>>> 
>>> The exception says that
>>> 
>>> Did you already try that?
>>> 
>>> 
>>> 
>>> --
>>> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/weird-client-failure-timeout-tp11201p11204.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/weird-client-failure-timeout-tp11201p11204.html>
>>> Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com <http://nabble.com/>.
>> 
> 


Mime
View raw message