kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Francesco vigotti (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6129) kafka issue when exposing through nodeport in kubernetes
Date Thu, 26 Oct 2017 10:34:00 GMT
Francesco vigotti created KAFKA-6129:
----------------------------------------

             Summary: kafka issue when exposing through nodeport in kubernetes
                 Key: KAFKA-6129
                 URL: https://issues.apache.org/jira/browse/KAFKA-6129
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.10.2.1
         Environment: kubernetes
            Reporter: Francesco vigotti
            Priority: Critical


I've started writing in this issue: https://issues.apache.org/jira/browse/KAFKA-2729
but then I'm going to open this new issue because I've probably found the cause in my kubernetes
setup, but In my opinion kubernetes did nothing wrong in his setup ( and all other application
works using the same nodeport redirection , ie: zookeeper )
kafka brokers fails , silently (randomly in multiple brokers setup)  and with a misleading
error from producer so I think that Kafka should be improved, providing more robust pre-startup
flight-checks and identifying/reporting the current issue 

After further investigation from my reply here https://issues.apache.org/jira/browse/KAFKA-2729
 with a minimum size cluster ( 1 zk + 1 kafka-broker ) I've found the problem, 
the problem is with kubernetes, ( I don't know why this issue appeared only now to me , if
something changed in recent kube-proxy versions or in kafka 0.10+ , or ... ) 
anyway my old kafka cluster started being underreplicated and return various problem , 

the problem happens when in kubernetes pods are created and redirected using a nodeport-service
( over a static ip in my case ) to expose kafka brokers from the host, when using hostNetwork
 ( so no redirection ) everything works, what is strange is that zookeeper instead works fine
with nodeport ( which create a redirection rule in iptables->nat->prerouting ) the only
application I've found problems with this kubernetes configuration is kafka,
what is weird is that kafka starts correctly without errors, but on multiple broker clusters
there are random issues, on single broker cluster instead the console-producer fails with
infinite looop of :

```
[2017-10-26 09:38:23,281] WARN Error while fetching metadata with correlation id 5 : {test6=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-10-26 09:38:23,383] WARN Error while fetching metadata with correlation id 6 : {test6=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-10-26 09:38:23,485] WARN Error while fetching metadata with correlation id 7 : {test6=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
```
, still no errors reported from broker or zookeeper,
Also I want to say that I've come across this discussion : 
             https://stackoverflow.com/questions/35788697/leader-not-available-kafka-in-console-producer

but the proposed solution for the host pod ( to allow self-resolving of advertised hostname)
didn't worked 
``` 
hostAliases:
      - ip: "127.0.0.1"
        hostnames:
        - "---myhosthostname---"
````






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message