kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Idriss Neumann (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-6191) Consumers are staying blocked on rebalancing
Date Thu, 09 Nov 2017 11:22:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Idriss Neumann updated KAFKA-6191:
----------------------------------
    Description: 
My logstash's consumers are staying blocked since couple days with a warn "group 'logstash'
is rebalancing". Here is the output of the "describe" command :

{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server host1:9092,host2:9092,host3:9092
--group logstash --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based
consumers).

Warning: Consumer group 'logstash' is rebalancing.

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID
                                      HOST                           CLIENT-ID
-                              -          -               -               -          logstash-4-faf1109b-8bec-41f7-b883-e422ebfa33a3
  /x.x.x.233                 logstash-4
-                              -          -               -               -          logstash-3-25b0380b-410e-42b6-8911-aab1a5435d48
  /x.x.x.233                 logstash-3
-                              -          -               -               -          logstash-2-9667ad4a-59aa-47e3-8633-51f3a9f13bb8
  /x.x.x.233                 logstash-2
-                              -          -               -               -          logstash-0-b205b42e-06ec-4d48-8edc-17c7d750fd66
  /x.x.x.233                 logstash-0
-                              -          -               -               -          logstash-1-990c6639-f8af-4476-a8cd-9a38c7ebd355
  /x.x.x.233                 logstash-1
rsyslog                        5          0               120815          120815     -   
                                             -                              -
collectd                       8          29828791        42691214        12862423   -   
                                             -                              -
collectd                       4          29796008        42898658        13102650   -   
                                             -                              -
rsyslog                        9          0               121227          121227     -   
                                             -                              -
collectd                       7          29752981        42739230        12986249   -   
                                             -                              -
rsyslog                        1          0               120739          120739     -   
                                             -                              -
collectd                       1          29779498        42783100        13003602   -   
                                             -                              -
rsyslog                        4          0               121258          121258     -   
                                             -                              -
collectd                       3          29839893        42688409        12848516   -   
                                             -                              -
rsyslog                        2          0               120306          120306     -   
                                             -                              -
collectd                       6          29795665        42742170        12946505   -   
                                             -                              -
rsyslog                        0          0               120045          120045     -   
                                             -                              -
collectd                       0          29805810        42883069        13077259   -   
                                             -                              -
rsyslog                        8          0               121075          121075     -   
                                             -                              -
rsyslog                        7          0               120463          120463     -   
                                             -                              -
rsyslog                        3          0               120773          120773     -   
                                             -                              -
collectd                       9          29803212        42631849        12828637   -   
                                             -                              -
collectd                       2          29810407        42654801        12844394   -   
                                             -                              -
rsyslog                        6          0               120381          120381     -   
                                             -                              -
collectd                       5          29803946        42793991        12990045   -   
                                             -                              -{code}

I've tried to restart all consumers, all brokers, same result. Moreover, if I change the group
id in the logstash conf, the consumers are working fine and consumes the messages from the
beginning:

{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server host1:9092,host2:9092,host3:9092
--group logstash_new --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based
consumers).


TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID
                                      HOST                           CLIENT-ID
rsyslog                        4          121276          121281          5          logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
collectd                       4          42899540        42899990        450        logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
rsyslog                        5          120831          120836          5          logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
collectd                       5          42794913        42795251        338        logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
rsyslog                        6          120397          120402          5          logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
collectd                       6          42743287        42743428        141        logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
rsyslog                        7          120484          120491          7          logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
collectd                       7          42740445        42740611        166        logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
rsyslog                        8          121089          121096          7          logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
collectd                       8          42692173        42692546        373        logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
rsyslog                        9          121245          121249          4          logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
collectd                       9          42632827        42633201        374        logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
rsyslog                        0          120066          120070          4          logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
collectd                       0          42884280        42884455        175        logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
rsyslog                        1          120763          120766          3          logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
collectd                       1          42784306        42784468        162        logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
rsyslog                        2          120325          120333          8          logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1
collectd                       2          42655705        42656061        356        logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1
rsyslog                        3          120789          120803          14         logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1
collectd                       3          42689329        42689665        336        logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1{code}

The logstash configuration :

{code}input {
  kafka {
    topics_pattern => 'collectd|rsyslog'
    group_id => 'logstash'
    bootstrap_servers => "host1:9092,host2:9092,host3:9092"
    consumer_threads => 5
    add_field => {
      from => "lst1"
    }
    codec => json
  }
}{code}

Thanks in advance for any help. 

  was:
My logstash's consumers are staying blocked since couple days with a warn "group 'logstash'
is rebalancing". Here is the output of the "describe" command :

{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server host1:9092,host2:9092,host3:9092
--group logstash --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based
consumers).

Warning: Consumer group 'logstash' is rebalancing.

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID
                                      HOST                           CLIENT-ID
-                              -          -               -               -          logstash-4-faf1109b-8bec-41f7-b883-e422ebfa33a3
  /x.x.x.233                 logstash-4
-                              -          -               -               -          logstash-3-25b0380b-410e-42b6-8911-aab1a5435d48
  /x.x.x.233                 logstash-3
-                              -          -               -               -          logstash-2-9667ad4a-59aa-47e3-8633-51f3a9f13bb8
  /x.x.x.233                 logstash-2
-                              -          -               -               -          logstash-0-b205b42e-06ec-4d48-8edc-17c7d750fd66
  /x.x.x.233                 logstash-0
-                              -          -               -               -          logstash-1-990c6639-f8af-4476-a8cd-9a38c7ebd355
  /x.x.x.233                 logstash-1
rsyslog                        5          0               120815          120815     -   
                                             -                              -
collectd                       8          29828791        42691214        12862423   -   
                                             -                              -
collectd                       4          29796008        42898658        13102650   -   
                                             -                              -
rsyslog                        9          0               121227          121227     -   
                                             -                              -
collectd                       7          29752981        42739230        12986249   -   
                                             -                              -
rsyslog                        1          0               120739          120739     -   
                                             -                              -
collectd                       1          29779498        42783100        13003602   -   
                                             -                              -
rsyslog                        4          0               121258          121258     -   
                                             -                              -
collectd                       3          29839893        42688409        12848516   -   
                                             -                              -
rsyslog                        2          0               120306          120306     -   
                                             -                              -
collectd                       6          29795665        42742170        12946505   -   
                                             -                              -
rsyslog                        0          0               120045          120045     -   
                                             -                              -
collectd                       0          29805810        42883069        13077259   -   
                                             -                              -
rsyslog                        8          0               121075          121075     -   
                                             -                              -
rsyslog                        7          0               120463          120463     -   
                                             -                              -
rsyslog                        3          0               120773          120773     -   
                                             -                              -
collectd                       9          29803212        42631849        12828637   -   
                                             -                              -
collectd                       2          29810407        42654801        12844394   -   
                                             -                              -
rsyslog                        6          0               120381          120381     -   
                                             -                              -
collectd                       5          29803946        42793991        12990045   -   
                                             -                              -{code}

If I change the group id in the logstash conf, the consumers are working fine and consumes
the messages from the beginning:

{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server host1:9092,host2:9092,host3:9092
--group logstash_new --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based
consumers).


TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID
                                      HOST                           CLIENT-ID
rsyslog                        4          121276          121281          5          logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
collectd                       4          42899540        42899990        450        logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
rsyslog                        5          120831          120836          5          logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
collectd                       5          42794913        42795251        338        logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327
  /x.x.x.228                 logstash-2
rsyslog                        6          120397          120402          5          logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
collectd                       6          42743287        42743428        141        logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
rsyslog                        7          120484          120491          7          logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
collectd                       7          42740445        42740611        166        logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991
  /x.x.x.228                 logstash-3
rsyslog                        8          121089          121096          7          logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
collectd                       8          42692173        42692546        373        logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
rsyslog                        9          121245          121249          4          logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
collectd                       9          42632827        42633201        374        logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877
  /x.x.x.228                 logstash-4
rsyslog                        0          120066          120070          4          logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
collectd                       0          42884280        42884455        175        logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
rsyslog                        1          120763          120766          3          logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
collectd                       1          42784306        42784468        162        logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd
  /x.x.x.228                 logstash-0
rsyslog                        2          120325          120333          8          logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1
collectd                       2          42655705        42656061        356        logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1
rsyslog                        3          120789          120803          14         logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1
collectd                       3          42689329        42689665        336        logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0
  /x.x.x.228                 logstash-1{code}

The logstash configuration :

{code}input {
  kafka {
    topics_pattern => 'collectd|rsyslog'
    group_id => 'logstash'
    bootstrap_servers => "host1:9092,host2:9092,host3:9092"
    consumer_threads => 5
    add_field => {
      from => "lst1"
    }
    codec => json
  }
}{code}

Thanks in advance for any help. 


> Consumers are staying blocked on rebalancing
> --------------------------------------------
>
>                 Key: KAFKA-6191
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6191
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.10.2.1
>            Reporter: Idriss Neumann
>
> My logstash's consumers are staying blocked since couple days with a warn "group 'logstash'
is rebalancing". Here is the output of the "describe" command :
> {code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server host1:9092,host2:9092,host3:9092
--group logstash --describe
> Note: This will only show information about consumers that use the Java consumer API
(non-ZooKeeper-based consumers).
> Warning: Consumer group 'logstash' is rebalancing.
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG       
CONSUMER-ID                                       HOST                           CLIENT-ID
> -                              -          -               -               -         
logstash-4-faf1109b-8bec-41f7-b883-e422ebfa33a3   /x.x.x.233                 logstash-4
> -                              -          -               -               -         
logstash-3-25b0380b-410e-42b6-8911-aab1a5435d48   /x.x.x.233                 logstash-3
> -                              -          -               -               -         
logstash-2-9667ad4a-59aa-47e3-8633-51f3a9f13bb8   /x.x.x.233                 logstash-2
> -                              -          -               -               -         
logstash-0-b205b42e-06ec-4d48-8edc-17c7d750fd66   /x.x.x.233                 logstash-0
> -                              -          -               -               -         
logstash-1-990c6639-f8af-4476-a8cd-9a38c7ebd355   /x.x.x.233                 logstash-1
> rsyslog                        5          0               120815          120815    
-                                                 -                              -
> collectd                       8          29828791        42691214        12862423  
-                                                 -                              -
> collectd                       4          29796008        42898658        13102650  
-                                                 -                              -
> rsyslog                        9          0               121227          121227    
-                                                 -                              -
> collectd                       7          29752981        42739230        12986249  
-                                                 -                              -
> rsyslog                        1          0               120739          120739    
-                                                 -                              -
> collectd                       1          29779498        42783100        13003602  
-                                                 -                              -
> rsyslog                        4          0               121258          121258    
-                                                 -                              -
> collectd                       3          29839893        42688409        12848516  
-                                                 -                              -
> rsyslog                        2          0               120306          120306    
-                                                 -                              -
> collectd                       6          29795665        42742170        12946505  
-                                                 -                              -
> rsyslog                        0          0               120045          120045    
-                                                 -                              -
> collectd                       0          29805810        42883069        13077259  
-                                                 -                              -
> rsyslog                        8          0               121075          121075    
-                                                 -                              -
> rsyslog                        7          0               120463          120463    
-                                                 -                              -
> rsyslog                        3          0               120773          120773    
-                                                 -                              -
> collectd                       9          29803212        42631849        12828637  
-                                                 -                              -
> collectd                       2          29810407        42654801        12844394  
-                                                 -                              -
> rsyslog                        6          0               120381          120381    
-                                                 -                              -
> collectd                       5          29803946        42793991        12990045  
-                                                 -                              -{code}
> I've tried to restart all consumers, all brokers, same result. Moreover, if I change
the group id in the logstash conf, the consumers are working fine and consumes the messages
from the beginning:
> {code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server host1:9092,host2:9092,host3:9092
--group logstash_new --describe
> Note: This will only show information about consumers that use the Java consumer API
(non-ZooKeeper-based consumers).
> TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG       
CONSUMER-ID                                       HOST                           CLIENT-ID
> rsyslog                        4          121276          121281          5         
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327   /x.x.x.228                 logstash-2
> collectd                       4          42899540        42899990        450       
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327   /x.x.x.228                 logstash-2
> rsyslog                        5          120831          120836          5         
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327   /x.x.x.228                 logstash-2
> collectd                       5          42794913        42795251        338       
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327   /x.x.x.228                 logstash-2
> rsyslog                        6          120397          120402          5         
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991   /x.x.x.228                 logstash-3
> collectd                       6          42743287        42743428        141       
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991   /x.x.x.228                 logstash-3
> rsyslog                        7          120484          120491          7         
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991   /x.x.x.228                 logstash-3
> collectd                       7          42740445        42740611        166       
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991   /x.x.x.228                 logstash-3
> rsyslog                        8          121089          121096          7         
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877   /x.x.x.228                 logstash-4
> collectd                       8          42692173        42692546        373       
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877   /x.x.x.228                 logstash-4
> rsyslog                        9          121245          121249          4         
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877   /x.x.x.228                 logstash-4
> collectd                       9          42632827        42633201        374       
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877   /x.x.x.228                 logstash-4
> rsyslog                        0          120066          120070          4         
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd   /x.x.x.228                 logstash-0
> collectd                       0          42884280        42884455        175       
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd   /x.x.x.228                 logstash-0
> rsyslog                        1          120763          120766          3         
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd   /x.x.x.228                 logstash-0
> collectd                       1          42784306        42784468        162       
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd   /x.x.x.228                 logstash-0
> rsyslog                        2          120325          120333          8         
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0   /x.x.x.228                 logstash-1
> collectd                       2          42655705        42656061        356       
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0   /x.x.x.228                 logstash-1
> rsyslog                        3          120789          120803          14        
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0   /x.x.x.228                 logstash-1
> collectd                       3          42689329        42689665        336       
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0   /x.x.x.228                 logstash-1{code}
> The logstash configuration :
> {code}input {
>   kafka {
>     topics_pattern => 'collectd|rsyslog'
>     group_id => 'logstash'
>     bootstrap_servers => "host1:9092,host2:9092,host3:9092"
>     consumer_threads => 5
>     add_field => {
>       from => "lst1"
>     }
>     codec => json
>   }
> }{code}
> Thanks in advance for any help. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message