flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: RocksDB error with flink 1.2.0
Date Tue, 02 May 2017 16:06:29 GMT
They can’t (with the current design of Flink) because each CEP pattern get’s executed by
a separate operator.

We could think about doing multiplexing of several patterns inside one operator. It’s what
I hinted at earlier as a possible solution when I mentioned that you could implement your
own operator that keeps track of the patterns and does the pattern matching.

Best,
Aljoscha
> On 2. May 2017, at 18:00, Elias Levy <fearsome.lucidity@gmail.com> wrote:
> 
> Any reason they can't share a single RocksDB state backend instance?
> 
> 
> On Fri, Apr 28, 2017 at 8:44 AM, Aljoscha Krettek <aljoscha@apache.org <mailto:aljoscha@apache.org>>
wrote:
> The problem here is that this will try to open 300 RocksDB instances on each of the TMs
(depending on how the parallelism is spread between the machines this could be more or less).
As the exception says, this will open too many files because each RocksDB instance has a directory
with several files in it.
> 
> One possible solution would be to increase the limit on open files but I don’t think
that opening 300 RocksDB instances on one machine is a good idea for any size of machine.
I think with this many patterns you could start thinking about writing the pattern matching
yourself and multiplexing the several patterns in one stateful function or operator.
> 
> @Stefan, what do you think about having this many Rocks instances?
> 
> 


Mime
View raw message