hadoop-zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anirban Roy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (ZOOKEEPER-496) zookeeper-tree utility for export, import and incremental updates
Date Wed, 18 Nov 2009 18:40:39 GMT

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Anirban Roy updated ZOOKEEPER-496:
----------------------------------

     Description: 
==========================================
zktreeutil - Zookeeper Tree Data Utility
Author: Anirban Roy
Organization: Yahoo Inc.
==========================================

zktreeutil program is intended to manage and manipulate zk-tree data quickly, effi-
ciently and with ease. The utility operates on free-form ZK-tree and hence can be used
for any cluster managed by Zookeeper. Here are the basic functionalities -

EXPORT: The whole/partial ZK-tree is exported into a XML file. This helps in
capturing a current snapshot of the data for backup/analysis. For a subtree
export, one need to specify the path to the ZK-subtree with proper option.

IMPORT: The ZK-tree can be imported from XML into ZK cluster. This helps in priming
the new ZK cluster with static configuration. The import can be non-intrusive by
making only the additions in the existing data. The import of subtree is also
possible by optionally providing the path to the ZK-subtree.

DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can ignore
some ZK-tree branches (possibly dynamic data) on reading the optional ignore flag
from XML file. Diffing on a ZK-subtree achieved by providing path to ZK-subtree with
diff command.

UPDATE: Make the incremental changes into the live ZK-tree from saved XML, essentia-
lly after running the diff.

DUMP: Dumps the ZK-tree on the standard output device reading either from live ZK
server or XML file. Like export, ZK-subtree can be dumped with optionaly
providing the path to the ZK-subtree, and till a certain depth of the (sub)tree.

The exported ZK data into XML file can be shortened by only keeping the static ZK
nodes which are required to prime a cluster. The dynamic zk nodes (created on-the-
fly) can be ignored by setting a 'ignore' attribute at the root node of the dynamic
subtree (see tests/zk_sample.xml), possibly deleting all inner ZK nodes under that.
Once ignored, the whole subtree is ignored during DIFF, UPDATE and WRITE.

  was:
I propose and plan to develop a ZK-tree utility for managing cluster data (ZK tree) quickly
and efficiently. The utility operates on free-form ZK-tree and hence can be used for any cluster
managed by Zookeeper. Here are the basic functionalities -

Export ZK-tree: The whole ZK-tree is exported into a XML file. This helps in capturing a current
snapshot of the data for backup/analysis. In addition, the data can be dumped as tree-like
text format which is easily readable.

Import ZK-tree: The ZK-tree can be imported from XML into ZK cluster. This helps in priming
the new ZK cluster with static configuration. The import can be non-intrusive by making only
the additions in the existing data.

Diff ZK-tree: Creates a diff between live ZK data vs data saved in XML file. Diff can ignore
some ZK-tree branches (possibly dynamic data) on reading the flag from XML file.

Update ZK-tree: Make the incremental changes into the live ZK-tree from saved XML, essentially
after running the diff.

Thanks
Anirban

    Release Note: 
==========================================
zktreeutil - Zookeeper Tree Data Utility
Author: Anirban Roy
Organization: Yahoo Inc.
==========================================

zktreeutil program is intended to manage and manipulate zk-tree data quickly, effi-
ciently and with ease. The utility operates on free-form ZK-tree and hence can be used
for any cluster managed by Zookeeper. Here are the basic functionalities -

EXPORT: The whole/partial ZK-tree is exported into a XML file. This helps in
capturing a current snapshot of the data for backup/analysis. For a subtree
export, one need to specify the path to the ZK-subtree with proper option.

IMPORT: The ZK-tree can be imported from XML into ZK cluster. This helps in priming
the new ZK cluster with static configuration. The import can be non-intrusive by
making only the additions in the existing data. The import of subtree is also
possible by optionally providing the path to the ZK-subtree.

DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can ignore
some ZK-tree branches (possibly dynamic data) on reading the optional ignore flag
from XML file. Diffing on a ZK-subtree achieved by providing path to ZK-subtree with
diff command.

UPDATE: Make the incremental changes into the live ZK-tree from saved XML, essentia-
lly after running the diff.

DUMP: Dumps the ZK-tree on the standard output device reading either from live ZK
server or XML file. Like export, ZK-subtree can be dumped with optionaly
providing the path to the ZK-subtree, and till a certain depth of the (sub)tree.

The exported ZK data into XML file can be shortened by only keeping the static ZK
nodes which are required to prime a cluster. The dynamic zk nodes (created on-the-
fly) can be ignored by setting a 'ignore' attribute at the root node of the dynamic
subtree (see tests/zk_sample.xml), possibly deleting all inner ZK nodes under that.
Once ignored, the whole subtree is ignored during DIFF, UPDATE and WRITE.

Pre-requisites
--------------
1. Linux system with 2.6.X kernel.
2. Zookeeper C client library >= 3.X.X
3. Development build libraries:
  a. libxml2 >= 2.7.3
  b. log4cxx >= 0.10.0

Build instructions
------------------
1. cd into this directory
2. autoreconf -if
3. ./configure
4. make
5. 'zktreeutil' binary created under src directory

Testing  and usage of zktreeutil
--------------------------------
1.  Run Zookeeper server locally on port 2181
2.  export LD_LIBRARY_PATH=../../c/.libs/:/usr/local/lib/
3.  ./src/zktreeutil --help # show help
4.  ./src/zktreeutil --zookeeper=localhost:2181 --import --xmlfile=tests/zk_sample.xml 2>/dev/null
            # import sample ZK tree
5.  ./src/zktreeutil --zookeeper=localhost:2181 --dump --path=/myapp/version-1.0 2>/dev/null
                  # dump Zk subtree
5.  ./src/zktreeutil --zookeeper=localhost:2181 --dump --depth=3 2>/dev/null          
                        # dump Zk tree till certain depth
6.  ./src/zktreeutil --xmlfile=zk_sample.xml -D 2>/dev/null                           
                        # dump the xml data
7.  Change zk_sample.xml with adding/deleting/chaging some nodes
8.  ./src/zktreeutil -z localhost:2181 -F -x zk_sample.xml -p /myapp/version-1.0/configuration
2>/dev/null     # take a diff of changes
9.  ./src/zktreeutil -z localhost:2181 -E 2>/dev/null > zk_sample2.xml             
                           # export the mofied ZK tree
10. ./src/zktreeutil -z localhost:2181 -U -x zk_sample.xml -p /myapp/version-1.0/distributions
2>/dev/null     # update with incr. changes
11. ./src/zktreeutil --zookeeper=localhost:2181 --import --force --xmlfile=zk_sample2.xml
2>/dev/null          # re-prime the ZK tree


  was:
==========================================
zktreeutil - Zookeeper Tree Data Utility
Author: Anirban Roy
Organization: Yahoo Inc.
==========================================

zktreeutil program is intended to manage and manipulate zk-tree data quickly, effi-
ciently and with ease.The utility operates on free-form ZK-tree and hence can be used
for any cluster managed by Zookeeper. Here are the basic functionalities -

EXPORT: The whole ZK-tree is exported into a XML file. This helps in capturing a cur-
rent snapshot of the data for backup/analysis. In addition, the data can be dumped
as tree-like text format which is easily readable.

IMPORT: The ZK-tree can be imported from XML into ZK cluster. This helps in priming
the new ZK cluster with static configuration. The import can be non-intrusive by
making only the additions in the existing data.

DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can ignore
some ZK-tree branches (possibly dynamic data) on reading the flag from XML file.

UPDATE: Make the incremental changes into the live ZK-tree from saved XML, essentia-
lly after running the diff.

DUMP: Dumps the ZK-tree on the standard output device reading either from live ZK
server or XML file.

The exported ZK data into XML file can be shortened by only keeping the static ZK
nodes which are required to prime a cluster. The dynamic zk nodes (created on-the-
fly) can be ignored by setting a 'ignore' attribute at the root node of the dynamic
subtree (see tests/zk_sample.xml), possibly deleting all inner ZK nodes under that.
Once ignored, the whole subtree is ignored during DIFF, UPDATE and WRITE.

Pre-requisites
--------------
1. Linux system with 2.6.X kernel.
2. Zookeeper C client library >= 3.X.X
3. Development build libraries:
  a. libxml2 >= 2.7.3
  b. log4cxx >= 0.10.0

Build instructions
------------------
1. cd into this directory
2. autoreconf -if
3. ./configure
4. make
5. 'zktreeutil' binary created under src directory

Testing zktreeutil
------------------
1.  Run Zookeeper server locally on port 2181
2.  export LD_LIBRARY_PATH=../../c/.libs/:/usr/local/lib/
3.  ./src/zktreeutil --help # show help
4.  ./src/zktreeutil --zookeeper=localhost:2181 --import --xmlfile=tests/zk_sample.xml 2>/dev/null
      # import sample ZK tree
5.  ./src/zktreeutil --zookeeper=localhost:2181 --dump 2>/dev/null                    
                  # dump on the terminal
6.  ./src/zktreeutil --xmlfile=zk_sample.xml -D 2>/dev/null                           
                  # dump the xml data
7.  Change zk_sample.xml with adding/deleting/chaging some nodes
8.  ./src/zktreeutil --zookeeper=localhost:2181 --diff --xmlfile=zk_sample.xml 2>/dev/null
              # take a diff of changes
9.  ./src/zktreeutil --zookeeper=localhost:2181 --export 2>/dev/null > zk_sample2.xml
                   # export the mofied ZK tree
10. ./src/zktreeutil --zookeeper=localhost:2181 --update --xmlfile=zk_sample.xml 2>/dev/null
            # update with incr. changes
11. ./src/zktreeutil --zookeeper=localhost:2181 --import --force --xmlfile=zk_sample2.xml
2>/dev/null    # re-prime the ZK tree



> zookeeper-tree utility for export, import and incremental updates
> -----------------------------------------------------------------
>
>                 Key: ZOOKEEPER-496
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-496
>             Project: Zookeeper
>          Issue Type: New Feature
>          Components: contrib
>         Environment: RHEL 4.6, libxml2
>            Reporter: Anirban Roy
>            Assignee: Anirban Roy
>             Fix For: 3.3.0
>
>         Attachments: zktreeutil.diff
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> ==========================================
> zktreeutil - Zookeeper Tree Data Utility
> Author: Anirban Roy
> Organization: Yahoo Inc.
> ==========================================
> zktreeutil program is intended to manage and manipulate zk-tree data quickly, effi-
> ciently and with ease. The utility operates on free-form ZK-tree and hence can be used
> for any cluster managed by Zookeeper. Here are the basic functionalities -
> EXPORT: The whole/partial ZK-tree is exported into a XML file. This helps in
> capturing a current snapshot of the data for backup/analysis. For a subtree
> export, one need to specify the path to the ZK-subtree with proper option.
> IMPORT: The ZK-tree can be imported from XML into ZK cluster. This helps in priming
> the new ZK cluster with static configuration. The import can be non-intrusive by
> making only the additions in the existing data. The import of subtree is also
> possible by optionally providing the path to the ZK-subtree.
> DIFF: Creates a diff between live ZK data vs data saved in XML file. Diff can ignore
> some ZK-tree branches (possibly dynamic data) on reading the optional ignore flag
> from XML file. Diffing on a ZK-subtree achieved by providing path to ZK-subtree with
> diff command.
> UPDATE: Make the incremental changes into the live ZK-tree from saved XML, essentia-
> lly after running the diff.
> DUMP: Dumps the ZK-tree on the standard output device reading either from live ZK
> server or XML file. Like export, ZK-subtree can be dumped with optionaly
> providing the path to the ZK-subtree, and till a certain depth of the (sub)tree.
> The exported ZK data into XML file can be shortened by only keeping the static ZK
> nodes which are required to prime a cluster. The dynamic zk nodes (created on-the-
> fly) can be ignored by setting a 'ignore' attribute at the root node of the dynamic
> subtree (see tests/zk_sample.xml), possibly deleting all inner ZK nodes under that.
> Once ignored, the whole subtree is ignored during DIFF, UPDATE and WRITE.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message