Hi,

Currently we need 75 Kafka partitions per topic and a parallelism of 75 to meet required performance, increasing the partitions and parallelism gives diminished returns

Currently the performance is approx. 1500 msg/s per core, having one pipeline (source, map, sink) deployed as one instance per core.

The Kafka source performance is not an issue. The map is very heavy (deserialization, validation) on rather complex Avro messages. Object reuse is enabled.

Ideally we would like to decouple Flink processing parallelism from Kafka partitions in a following manner:
AFAIK keyBy, partitionCustom will distribute messages over the network and rescale has no affinity for message identity.

Am I missing something obvious?

Cheers,

Sanne