zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@apache.org>
Subject Re: Messages intermittently get lost
Date Wed, 30 Nov 2016 23:41:24 GMT
Hey Martin,

Do you want to create a jira for this and report the issue?

-Flavio

> On 30 Nov 2016, at 18:33, Martin Gainty <mgainty@hotmail.com> wrote:
> 
> a shock when a zk script goes fubar
> 
> 
> zookeeper devs can we get some help for sh zkServer.sh status?
> 
> 
> Thanks!
> 
> Martin
> ______________________________________________
> 
> 
> 
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> Hi Martin, makes sense.
> 
> 
> When I SSH into all 3 of my ZK nodes and run:
> 
> 
> sh zkServer.sh status
> 
> 
> All three of them give me the following output:
> 
> 
> JMX enabled by default
> 
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting
"fi")
> 
> 
> Looks like a bug in the ZK shell script?
> 
> 
> Best,
> 
> Zac
> 
> ________________________________
> From: Martin Gainty <mgainty@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> Hi Zach
> 
> 
> we dont know whats causing this intermittent problem..so lets Divide and Conquer each
part of this problem individually starting at the source of the data feeds
> 
> 
> Let us eliminate any potential problem with feeds from external sources
> 
> 
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
> 
> 
> Pingback when you have verifiable results from zookeeper feeds
> 
> 
> Thanks
> 
> Martin
> ______________________________________________
> 
> 
> 
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer
fail to be received by a Kafka consumer?
> 
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> Thanks Martin, I will look at those links.
> 
> 
> But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why?
What is it about my problem description that makes you think this is an issue with ZooKeeper?
> 
> ________________________________
> From: Martin Gainty <mgainty@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> 
> 
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> Hi Martin,
> 
> 
> My server.properties looks like this:
> 
> 
> listeners=PLAINTEXT://0.0.0.0:9092
> 
> advertised.host.name=<SERVER_PUBLIC_DNS>
> 
> broker.id=2
> 
> port=9092
> 
> num.partitions=4
> 
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
> 
> MG>can you check status for each ZK Node in the quorum?
> 
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
> 
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> 
> 
> 
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> 
> 
> 
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
> 
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
> 
> 
> 
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> 
> 
> 
> 
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> 
> 
> 
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> 
> 
> 
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
> 
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
> 
> 
> 
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> 
> 
> 
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
> 
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
> 
> 
> 
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
> 
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
> 
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
> 
> 
> 
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
> 
> IBM - United States<http://www.ibm.com/>
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
> 
> 
> 
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
> 
> 
> 
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
> 
> 
> 
> MG>*greetings from Plimoth Mass*
> 
> MG>M
> 
> 
> num.network.threads=3
> 
> num.io.threads=8
> 
> socket.send.buffer.bytes=102400
> 
> socket.receive.buffer.bytes=102400
> 
> log.dirs=/tmp/kafka-logs
> 
> num.recovery.threads.per.data.dir=1
> 
> log.retention.hours=168
> 
> log.segment.bytes=1073741824
> 
> log.retention.check.interval.ms=300000
> 
> zookeeper.connection.timeout.ms=6000
> 
> delete.topic.enable=true
> 
> auto.leader.rebalance.enable=true
> 
> 
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and
<SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running
on.
> 
> 
> Anything look incorrect to you?
> 
> 
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long
holiday when I'm dead!
> 
> 
> Thanks for any-and-all input here!
> 
> 
> Best,
> 
> Zac
> 
> 
> ________________________________
> From: Martin Gainty <mgainty@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> Hi Zach
> 
> 
> there is a rumour that today thursday is a holiday?
> 
> in server.properties how are you configuring your server?
> 
> 
> specifically what are these attributes?
> 
> 
> num.network.threads=
> 
> 
> num.io.threads=
> 
> 
> socket.send.buffer.bytes=
> 
> 
> socket.receive.buffer.bytes=
> 
> 
> socket.request.max.bytes=
> 
> 
> num.partitions=
> 
> 
> num.recovery.threads.per.data.dir=
> 
> 
> ?
> 
> Martin
> ______________________________________________
> 
> 
> 
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
> 
> Anybody?!? This is very disconcerting!
> 
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
> 
> I am playing around with Kafka and have a simple setup:
> 
> 
> * 1-node Kafka (Ubuntu) server
> 
> * 3-node ZK cluster (each on their own Ubuntu server)
> 
> 
> I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that
ships with the distribution.
> 
> 
> I'd say about 20% of the messages I send via the producer never get consumed by the Scala
process (which is running continuously). No errors on either side (producer or consumer):
the producer sends, and, nothing...
> 
> 
> Any ideas as to what might be going on here, or how I could start troubleshooting?
> 
> 
> Thanks!


Mime
View raw message