Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of rakeshr@huawei.com designates
 119.145.14.65 as permitted sender)
From: Rakesh R <rakeshr@huawei.com>
To: "user@zookeeper.apache.org" <user@zookeeper.apache.org>
CC: "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>
Subject: RE: Can ZK be used for my use case?
Thread-Topic: Can ZK be used for my use case?
Thread-Index: AQHOqlfBPj3FkgxQ8UqDN0i24tlM5Zm3jnWAgACeMpA=
Date: Fri, 6 Sep 2013 05:30:05 +0000
Message-ID: 
 <C2496325850AA74C92AAF83AA9662D2631B84488@szxeml561-mbx.china.huawei.com>
References: <1378391817809-7579049.post@n2.nabble.com>
 <228CE998-4320-46D7-AE06-64CEF06CBD4D@gmail.com>
In-Reply-To: <228CE998-4320-46D7-AE06-64CEF06CBD4D@gmail.com>
Accept-Language: en-US, zh-CN
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Tavi,

If I understood the usecase correctly, you have different types of files wh=
ich is present on a FTP server. There are 300 clients which would be intere=
sted on the file modifications and watching these files, when there is a ve=
rsion change they would access those files and will act upon.

If this is the case, you can create set of zNodes which will be representin=
g each files like /parentNode/file1, /parentNode/file2, /parentNode/file3..=
. etc.
Interested Clients can add DataWatchers to these files and write your clien=
t side logic on recieving the watches notification.
For example,=20
Client1 -> /parentNode/file1, /parentNode/file2
Client2 -> /parentNode/file2, /parentNode/file3
Client3 -> /parentNode/file4, /parentNode/file1

Say Client1 wants to modify /parentNode/file2, first he should acquire a lo=
ck(please see distributed lock recipe using zookeeper) on the file and afte=
r modification should add metadata (host:port, or any other unique key to g=
et the trace, who has done the changes) on the zNode /parentNode/file2 and =
release the lock. After updating the metadata of /parentNode/file2, Client1=
 and Client3 would get the data watches notification and can act accordingl=
y.

All your clients can add child watches to the /parentNode to know the file =
addition/creation on the FTPServer and decide whether to add DataWatchers o=
r not.

Always keep this in mind: Default zNode data size is 1MB and recommended to=
 keep lesser data on zNode(its tunable/configurable parameter, user can dec=
ide).
ZooKeeper is designed as a high read, high throughput system for small data=
. It is not designed as a large data store to hold very large data values. =
As such this 1MB value is a default config option and can be overridden.  I=
t is not advised to do so - but increasing the size a little bit will proba=
bly not damage your system (it all depends on your unique access patterns a=
nd these changes should be made with care and at your own risk).

Also, network fluctuations could affect the watch notifications, there coul=
d be high chance of missing watch notification when it involves network flu=
ctuations and should have better handling of ZooKeeper connection events.


BTW, How many files would be present in the FTPServer?.  Is 300 clients fix=
ed always or dynamically grows?

What if, the client missed one of the version change notification and would=
 like to know the frequency of changing the same file again and again?

-Rakesh

-----Original Message-----
From: Martin Kou [mailto:bitanarch@gmail.com]=20
Sent: 06 September 2013 09:23
To: user@zookeeper.apache.org
Cc: zookeeper-user@hadoop.apache.org
Subject: Re: Can ZK be used for my use case?

How large are your files? ZooKeeper is generally not designed for large siz=
e storage. Also, it doesn't provide guarantees of watchers being called whe=
n network outage is involved.

Sent from my iPhone

> On Sep 5, 2013, at 7:36 AM, Tavi <tavi1976@yahoo.com> wrote:
>=20
> Hi everyone,
>=20
> I have a web application who generates different types of files (at=20
> different version) on the server.  Those files must be transferred at=20
> 300 clients (watchers) and, after transfer, each client must modify=20
> every file for his purpose.  The "client" is a simple java stand alone=20
> application installed on my user PC.
> Today my users must download manually their files from an FTP server=20
> and they must use a java library to convert their files.  Of course, I=20
> don't know if they do this when the file version changes, I don't have=20
> any trace who did it ...
>=20
> My idea is to use a ZK server with 300 clients, each client will be=20
> monitoring a specific namespace for changes (for example : client 1=20
> will monitor /app1 and app2, client 2 -> app 2 only ...).  Every=20
> namespace will contain the name of the file, his location, his version=20
> ... When the file version will change, the java client will use the=20
> information to retrieve and to modify the file from my central FTP.=20
> "My leader" needs to know which user is connected and he must=20
> synchronize the downloads to avoid network bottlenecks.
>=20
> Does ZK fits in this scenario?
>=20
> Thanks for your time,
>  Tavi
>=20
>=20
>=20
>=20
> --
> View this message in context:=20
> http://zookeeper-user.578899.n2.nabble.com/Can-ZK-be-used-for-my-use-c
> ase-tp7579049.html Sent from the zookeeper-user mailing list archive=20
> at Nabble.com.