zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Gainty <mgai...@hotmail.com>
Subject Setting ZOO_DATADIR
Date Fri, 02 Dec 2016 03:28:49 GMT
apparently when ZOO_DATADIR is empty mkdir fubars mkdir ''

when ZOO_DATADIR is empty add this export ZOO_DATADIR="zoo_datadir"


if [ -z "$ZOOPIDFILE" ]; then
    ZOO_DATADIR="$(grep "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//')"
    if [ ! -d "$ZOO_DATADIR" ]; then
        echo "ZOO_DATADIR="

        export ZOO_DATADIR="zoo_datadir"
        echo $ZOO_DATADIR

        mkdir -p "$ZOO_DATADIR"
    fi

HTH
M-


________________________________
From: Zac Harvey <zac.harvey@welltok.com>
Sent: Thursday, December 1, 2016 11:03 AM
To: Martin Gainty; user@zookeeper.apache.org
Subject: Re: Messages intermittently get lost


Thanks everyone


The script can be found like so:


wget http://apache.spinellicreations.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

tar xfz zookeeper-3.4.6.tar.gz

cd zookeeper-3.4.6/bin

zkServer.sh


Let me know if you have any other questions/concerns!


Best,

Zac

________________________________
From: Martin Gainty <mgainty@hotmail.com>
Sent: Wednesday, November 30, 2016 7:57:37 PM
To: user@zookeeper.apache.org; Zac Harvey
Subject: Re: Messages intermittently get lost


absolutely..lets get this resolved asap


Zac...can you send me the abending script?


Molte Grazie Flavio!

Martini
______________________________________________



________________________________
From: Flavio Junqueira <fpj@apache.org>
Sent: Wednesday, November 30, 2016 6:41 PM
To: user@zookeeper.apache.org
Cc: users@kafka.apache.org
Subject: Re: Messages intermittently get lost

Hey Martin,

Do you want to create a jira for this and report the issue?

-Flavio

> On 30 Nov 2016, at 18:33, Martin Gainty <mgainty@hotmail.com> wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting
"fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> ________________________________
> From: Martin Gainty <mgainty@hotmail.com>
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and Conquer each
part of this problem individually starting at the source of the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a Kafka producer
fail to be received by a Kafka consumer?
>
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why?
What is it about my problem description that makes you think this is an issue with ZooKeeper?
>
> ________________________________
> From: Martin Gainty <mgainty@hotmail.com>
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=<SERVER_PUBLIC_DNS>
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
> http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html>
www.ibm.com
Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.



> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> www.ibm.com<http://www.ibm.com>
[http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>

IBM - United States<http://www.ibm.com/>
www.ibm.com
For more than a century IBM has been dedicated to every client's success and to creating innovations
that matter for the world



> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
>
>
>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
>
>
>
> www.ibm.com<http://www.ibm.com>
> [http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png]<http://www.ibm.com/>
>
> IBM - United States<http://www.ibm.com/>
> www.ibm.com<http://www.ibm.com>
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
>
>
>
> For more than a century IBM has been dedicated to every client's success and to creating
innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper.
>
>
>
> MG>*greetings from Plimoth Mass*
>
> MG>M
>
>
> num.network.threads=3
>
> num.io.threads=8
>
> socket.send.buffer.bytes=102400
>
> socket.receive.buffer.bytes=102400
>
> log.dirs=/tmp/kafka-logs
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> auto.leader.rebalance.enable=true
>
>
> Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and
<SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running
on.
>
>
> Anything look incorrect to you?
>
>
> And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long
holiday when I'm dead!
>
>
> Thanks for any-and-all input here!
>
>
> Best,
>
> Zac
>
>
> ________________________________
> From: Martin Gainty <mgainty@hotmail.com>
> Sent: Thursday, November 24, 2016 9:03:33 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> there is a rumour that today thursday is a holiday?
>
> in server.properties how are you configuring your server?
>
>
> specifically what are these attributes?
>
>
> num.network.threads=
>
>
> num.io.threads=
>
>
> socket.send.buffer.bytes=
>
>
> socket.receive.buffer.bytes=
>
>
> socket.request.max.bytes=
>
>
> num.partitions=
>
>
> num.recovery.threads.per.data.dir=
>
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Thursday, November 24, 2016 7:05 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Anybody?!? This is very disconcerting!
>
> ________________________________
> From: Zac Harvey <zac.harvey@welltok.com>
> Sent: Wednesday, November 23, 2016 5:07:45 AM
> To: users@kafka.apache.org
> Subject: Messages intermittently get lost
>
> I am playing around with Kafka and have a simple setup:
>
>
> * 1-node Kafka (Ubuntu) server
>
> * 3-node ZK cluster (each on their own Ubuntu server)
>
>
> I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that
ships with the distribution.
>
>
> I'd say about 20% of the messages I send via the producer never get consumed by the Scala
process (which is running continuously). No errors on either side (producer or consumer):
the producer sends, and, nothing...
>
>
> Any ideas as to what might be going on here, or how I could start troubleshooting?
>
>
> Thanks!


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message