flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From fhueske <...@git.apache.org>
Subject [GitHub] flink issue #3590: [FLINK-5654] - Add processing time OVER RANGE BETWEEN x P...
Date Sun, 26 Mar 2017 19:04:32 GMT
Github user fhueske commented on the issue:

    https://github.com/apache/flink/pull/3590
  
    Hi @rtudoran,
    
    thanks for doing the benchmark and posting the numbers! The recommended state backend
for production settings is the RocksDB backend (see [production-readiness docs](https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/production_ready.html#choice-of-state-backend)).
The in-memory backends store state as objects on the heap and can easily kill the JVM with
an OutOfMemoryError. Also the in-memory backends do not de/serialize data, so there is not
an actual advantage is using the MapState which was mainly motivated by the reduced serialization
effort. There are plans to implement a state backend using managed memory (similar to the
batch algorithms). This backend would also serialize and deserialize data to/from pre-allocated
byte arrays. So optimizing for de/serialization makes sense, IMO.
    
    The `KeyedOneInputStreamOperatorTestHarness` is part of the `flink-streaming-java` test-jar
artifact. This is added by adding the following dependency to the `flink-table` `pom.xml`.
    
    ```<dependency>
    	<groupId>org.apache.flink</groupId>
    	<artifactId>flink-streaming-java_2.10</artifactId>
    	<version>${project.version}</version>
    	<type>test-jar</type>
    	<scope>test</scope>
    </dependency>
    ```
    
    I'll have a detailed look at your PR tomorrow.
    Best, Fabian


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message