apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siyuan Hua (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2169) KafkaInputOperator: Dynamic partition is not working in case of ONE_TO_MANY partition strategy
Date Wed, 27 Jul 2016 20:55:20 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396353#comment-15396353
] 

Siyuan Hua commented on APEXMALHAR-2169:
----------------------------------------

[~chaithu] 

Can you elaborate more on this ticket? Is it dynamic partition based on throughput or metadata
change or both?
If you want dynamic partition happen because more kafka partitions are added, I think the
code should jump into line 599 where newWaitingPartition should be non-empty, right?



> KafkaInputOperator: Dynamic partition is not working in case of ONE_TO_MANY partition
strategy
> ----------------------------------------------------------------------------------------------
>
>                 Key: APEXMALHAR-2169
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2169
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: Chaitanya
>            Assignee: Chaitanya
>
> Dynamic Partition is not working in case of ONE_TO_MANY partition strategy.
> Affected Operator: AbstractKafkaInputOperator (0.8 version)
> Steps to reproduce: 
> (1) Created a topic with 3 partitions
> (2) Created an application as  KAFKA -> Console with below configuration:
>        strategy : one_to_many
>        initialPartitionCount: 2
> (3) Launched the above application.
> (4) After some time, re-partition the topic to 5
> Observations:
> (1) Operator re-partitioning is not happened.
> (2) Operator is not emitting the messages.
> (3) Following warning messages in log:
> INFO com.datatorrent.contrib.kafka.AbstractKafkaInputOperator: [ONE_TO_MANY]: Repartition
the operator(s) under 9223372036854775807 msgs/s and 9223372036854775807 bytes/s hard limit
> WARN com.datatorrent.stram.plan.physical.PhysicalPlan: Empty partition list after repartition:
OperatorMeta{name=Input, operator=com.datatorrent.contrib.kafka.KafkaSinglePortByteArrayInputOperator@1b822fcc,
attributes={Attribute{defaultValue=1024, name=com.datatorrent.api.Context.OperatorContext.MEMORY_MB,
codec=com.datatorrent.api.StringCodec$Integer2String@4682eba5}=1024}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message