zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Han <h...@cloudera.com>
Subject Re: Config options not being loaded?
Date Thu, 21 Jul 2016 23:34:00 GMT
>> using zookeeper only in memory

I don't think you can completely disable persistency - ZK will at least
write transaction logs and snap shot to disk in any case. There is
ZOOKEEPER-866 <https://issues.apache.org/jira/browse/ZOOKEEPER-866> relates
to disable persistency completely for ZK, which is probably what you are
looking for ultimately, but that work is not done yet.

>> what would you suggest?

I think your original config looks OK, in particular:
- 'forceSync=no' will make transaction log write async.
- Set snapCount to a really large number to disable snap shot as it seems
you don't have durable requirement and not care about recovery time.

You might also want to clarify / provide more information regarding your
use case (e.g. work load type, read heavy or write heavy, etc), and maybe
some users here that had similar use cases can provide their insights.


On Thu, Jul 21, 2016 at 2:35 PM, André Mantas <andremantas7@gmail.com>
wrote:

> Ok thanks I'll check it out. As for my other question of using zookeeper
> only in memory, what would you suggest?
>
> Thanks for the help so far.
>
> A 10:18PM Qui, 21 de Jul de 2016, Michael Han <hanm@cloudera.com>
> escreveu:
>
> > I think it is not guaranteed that we can find all configuration options
> in
> > server logs because ZK may not log all available configuration options.
> > Another idea to verify the configuration options are loaded with the
> value
> > you specified is to examine the Java system properties of the JVM ZK
> > process, because all the configuration options are loaded as Java system
> > properties. As an example, skipACL will also be presented as
> > zookeeper.skipACL property.
> >
> > On Thu, Jul 21, 2016 at 2:07 PM, André Mantas <andremantas7@gmail.com>
> > wrote:
> >
> > > Thanks for the reply. Indeed I was able to see the skip ACL config in
> the
> > > server log but not any of the other 3 configs. Was it because I
> > specified a
> > > wrong value for these configs maybe?
> > >
> > > A 9:56PM Qui, 21 de Jul de 2016, Michael Han <hanm@cloudera.com>
> > escreveu:
> > >
> > > > The fact that 'conf' does not print out all configuration options
> > encoded
> > > > in zoo.cfg does not necessarily mean these options are not loaded in
> > ZK.
> > > In
> > > > face the options 'conf' will print out is pretty limited and you can
> > > check
> > > > those options in ZooKeeperServerConf.java
> > > > <
> > > >
> > >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/ZooKeeperServerConf.java#L26
> > > > >.
> > > > It'll be good to make this class up to date with regards to latest
> > > > available configuration options in ZK but most if not all
> configuration
> > > > options could also be validated in server logs (like the skipACL one
> > > here).
> > > >
> > > > On Thu, Jul 21, 2016 at 11:55 AM, André Mantas <
> andremantas7@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Follow up:
> > > > > With start-foreground I'm able to see that skipACL=yes was
> effective
> > > > (full
> > > > > output in the end of the email):
> > > > >
> > > > > 2016-07-21 20:36:19,433 [myid:] - INFO
> [main:PrepRequestProcessor@87
> > ]
> > > -
> > > > > zookeeper.skipACL=="yes", ACL checks will be skipped
> > > > >
> > > > > What is the problem with the other config values? I'm trying to
> boost
> > > > > performance by disabling the persistency of zookeeper. I'm only
> > > > interested
> > > > > in using zookeeper in memory database (no need for data persistency
> > > after
> > > > > reboot).
> > > > >
> > > > > Any advice on how to achieve this?
> > > > >
> > > > > Thanks
> > > > >
> > > > > -----
> > > > >
> > > > > ~/amantas/zookeeper-3.4.8# ./bin/zkServer.sh start-foreground
> > > > > ZooKeeper JMX enabled by default
> > > > > Using config: /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > 2016-07-21 20:36:15,060 [myid:] - INFO  [main:QuorumPeerConfig@103
> ]
> > -
> > > > > Reading configuration from:
> > > > > /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > 2016-07-21 20:36:15,069 [myid:] - INFO
> > [main:DatadirCleanupManager@78
> > > ]
> > > > -
> > > > > autopurge.snapRetainCount set to 3
> > > > > 2016-07-21 20:36:15,069 [myid:] - INFO
> > [main:DatadirCleanupManager@79
> > > ]
> > > > -
> > > > > autopurge.purgeInterval set to 0
> > > > > 2016-07-21 20:36:15,069 [myid:] - INFO
> > [main:DatadirCleanupManager@101
> > > ]
> > > > -
> > > > > Purge task is not scheduled.
> > > > > 2016-07-21 20:36:15,071 [myid:] - WARN  [main:QuorumPeerMain@113]
> -
> > > > Either
> > > > > no config or no quorum defined in config, running  in standalone
> mode
> > > > > 2016-07-21 20:36:15,093 [myid:] - INFO  [main:QuorumPeerConfig@103
> ]
> > -
> > > > > Reading configuration from:
> > > > > /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > 2016-07-21 20:36:15,094 [myid:] - INFO
> [main:ZooKeeperServerMain@95
> > ]
> > > -
> > > > > Starting server
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18
> GMT
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:host.name=t8.wan
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.version=1.8.0_91
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.vendor=Oracle Corporation
> > > > > 2016-07-21 20:36:15,105 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > >
> > > > >
> > > >
> > >
> >
> environment:java.class.path=/root/amantas/zookeeper-3.4.8/bin/../build/classes:/root/amantas/zookeeper-3.4.8/bin/../build/lib/*.jar:/root/amantas/zookeeper-3.4.8/bin/../lib/slf4j-log4j12-1.6.1.jar:/root/amantas/zookeeper-3.4.8/bin/../lib/slf4j-api-1.6.1.jar:/root/amantas/zookeeper-3.4.8/bin/../lib/netty-3.7.0.Final.jar:/root/amantas/zookeeper-3.4.8/bin/../lib/log4j-1.2.16.jar:/root/amantas/zookeeper-3.4.8/bin/../lib/jline-0.9.94.jar:/root/amantas/zookeeper-3.4.8/bin/../zookeeper-3.4.8.jar:/root/amantas/zookeeper-3.4.8/bin/../src/java/lib/*.jar:/root/amantas/zookeeper-3.4.8/bin/../conf:
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > >
> > > > >
> > > >
> > >
> >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.io.tmpdir=/tmp
> > > > > 2016-07-21 20:36:15,106 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:java.compiler=<NA>
> > > > > 2016-07-21 20:36:15,108 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:os.name=Linux
> > > > > 2016-07-21 20:36:15,108 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:os.arch=amd64
> > > > > 2016-07-21 20:36:15,108 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:os.version=4.4.0-21-generic
> > > > > 2016-07-21 20:36:15,109 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:user.name=root
> > > > > 2016-07-21 20:36:15,109 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:user.home=/root
> > > > > 2016-07-21 20:36:15,109 [myid:] - INFO  [main:Environment@100] -
> > > Server
> > > > > environment:user.dir=/root/amantas/zookeeper-3.4.8
> > > > > 2016-07-21 20:36:15,118 [myid:] - INFO  [main:ZooKeeperServer@787]
> -
> > > > > tickTime set to 2000
> > > > > 2016-07-21 20:36:15,118 [myid:] - INFO  [main:ZooKeeperServer@796]
> -
> > > > > minSessionTimeout set to -1
> > > > > 2016-07-21 20:36:15,118 [myid:] - INFO  [main:ZooKeeperServer@805]
> -
> > > > > maxSessionTimeout set to -1
> > > > > 2016-07-21 20:36:15,132 [myid:] - INFO
> [main:NIOServerCnxnFactory@89
> > ]
> > > -
> > > > > binding to port 0.0.0.0/0.0.0.0:2181
> > > > > 2016-07-21 20:36:19,433 [myid:] - INFO
> [main:PrepRequestProcessor@87
> > ]
> > > -
> > > > > zookeeper.skipACL=="yes", ACL checks will be skipped
> > > > >
> > > > > André Mantas <andremantas7@gmail.com> escreveu no dia quinta,
> > > 21/07/2016
> > > > > às
> > > > > 03:28:
> > > > >
> > > > > > Hi all.
> > > > > >
> > > > > > I'm trying to experiment with some config options found in
> > > > > >
> > > > >
> > > >
> > >
> >
> https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_configuration
> > > > > but
> > > > > > they are not being loaded/used by the zookeeper server.
> > > > > >
> > > > > > I've modified my config file 'zoo.cfg' to:
> > > > > >
> > > > > > # The number of milliseconds of each tick
> > > > > > tickTime=2000
> > > > > > # The number of ticks that the initial
> > > > > > # synchronization phase can take
> > > > > > initLimit=10
> > > > > > # The number of ticks that can pass between
> > > > > > # sending a request and getting an acknowledgement
> > > > > > syncLimit=5
> > > > > > # the directory where the snapshot is stored.
> > > > > > # do not use /tmp for storage, /tmp here is just
> > > > > > # example sakes.
> > > > > > dataDir=/root/amantas/zookeeper-3.4.8/data
> > > > > > # the port at which the clients will connect
> > > > > > clientPort=2181
> > > > > > # Requires updates to be synced to media of the transaction
log
> > > before
> > > > > > finishing processing the update.
> > > > > > # If this option is set to no, ZooKeeper will not require updates
> > to
> > > be
> > > > > > synced to the media.
> > > > > > forceSync=no
> > > > > > # Skips ACL checks. This results in a boost in throughput, but
> > opens
> > > up
> > > > > > full access to the data tree to everyone.
> > > > > > skipACL=yes
> > > > > > # The observers now log transaction and write snapshot to disk
by
> > > > default
> > > > > > like the participants.
> > > > > > # This reduces the recovery time of the observers on restart.
> > > > > > # Set to "false" to disable this feature. Default is "true"
> > > > > > syncEnabled=false
> > > > > > # ZooKeeper logs transactions to a transaction log.
> > > > > > # After snapCount transactions are written to a log file a
> snapshot
> > > is
> > > > > > started and a new transaction log file is created.
> > > > > > # The default snapCount is 100,000
> > > > > > snapCount=1000000
> > > > > >
> > > > > > Then, restarted the zookeeper server:
> > > > > >
> > > > > > ~/amantas/zookeeper-3.4.8# ./bin/zkServer.sh stop
> > > > > > ZooKeeper JMX enabled by default
> > > > > > Using config: /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > > Stopping zookeeper ... STOPPED
> > > > > >
> > > > > > ~/amantas/zookeeper-3.4.8# ./bin/zkServer.sh start
> > > > > > ZooKeeper JMX enabled by default
> > > > > > Using config: /root/amantas/zookeeper-3.4.8/bin/../conf/zoo.cfg
> > > > > > Starting zookeeper ... STARTED
> > > > > >
> > > > > > However, when listing the server configuration I don't see some
> of
> > > the
> > > > > > config params:
> > > > > >
> > > > > > ~/amantas/zookeeper-3.4.8# nc localhost 2181
> > > > > > conf
> > > > > > clientPort=2181
> > > > > > dataDir=/root/amantas/zookeeper-3.4.8/data/version-2
> > > > > > dataLogDir=/root/amantas/zookeeper-3.4.8/data/version-2
> > > > > > tickTime=2000
> > > > > > maxClientCnxns=60
> > > > > > minSessionTimeout=4000
> > > > > > maxSessionTimeout=40000
> > > > > > serverId=0
> > > > > >
> > > > > > What am I doing wrong?
> > > > > >
> > > > > > Thanks in advance!
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >
>



-- 
Cheers
Michael.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message