flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Kim <david....@braintreepayments.com>
Subject Re: Flink Kafka Unused Properties Log Warnings
Date Wed, 01 Jun 2016 19:19:04 GMT
Awesome, thanks Robert!

On Wed, Jun 1, 2016 at 1:12 PM Robert Metzger <rmetzger@apache.org> wrote:

> Hi Kim,
>
> I agree with you. We should definitively filter out custom Flink
> properties. I've filed a JIRA for this:
> https://issues.apache.org/jira/browse/FLINK-4004
> Thanks for the report!
>
> Regards,
> Robert
>
>
> On Wed, Jun 1, 2016 at 6:42 PM, David Kim <david.kim@braintreepayments.com
> > wrote:
>
>> Hello!
>>
>> Using Flink 1.0.3.
>>
>> This is cosmetic but will help clean up logging I think.
>>
>> The Apache *KafkaConsumer* logs a warning [1] for any unused properties.
>> This is great in case the developer has a typo or should clean up any
>> unused keys.
>>
>> Flink's Kafka consumer and producer have some custom properties and it
>> passes the entire properties map into the Apache Kafka consumer and
>> producer. Custom properties such as "flink.poll-timeout" make it through
>> and it triggers a warning.
>>
>> For the sake of cleaning up logging and removing false positives, I think
>> it might make sense for the FlinkKafkaConsumer09 [2] and
>> FlinkKafkaProducerBase [3] to filter out custom flink properties.
>>
>> Let me know if this sounds reasonable!
>>
>> Thanks!
>> David
>>
>> [1]
>> https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L639
>> [2]
>> https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java#L186
>> [3]
>> https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L153
>>
>>
>>
>

Mime
View raw message