hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: ConnectionLoss (node too big?)
Date Wed, 03 Jun 2009 16:46:40 GMT
Agree, created a new JIRA for this:

See the following JIRA for one example why not to do this:

In general you don't want to create large node sizes since all of the 
data/nodes are stored in memory by all of the servers. The latency issue 
is also a factor. However if you are storing a  handful of nodes in the 
cluster then obv these aren't much of a problem (could bite you at some 
point in the future though if you start using ZK more...) In general we 
advise ppl to store "tokens" in ZK, so perhaps you might store the 7mb 
of data in a data store (filesystem?), and use ZK to coordinate access 
to that data (this is similar for example to how AWS does things with S3 
and SQS, SQS has a limit of 8k iirc, so you store the task in SQS which 
includes a pointer (url) to the data to be acted upon in S3...)


Eric Bowman wrote:
> Ted Dunning wrote:
>> Isn't the max file size a megabyte?
>> On Wed, Jun 3, 2009 at 9:01 AM, Eric Bowman <ebowman@boboco.ie> wrote:
>>> On the client, I see this when trying to write a node with 7,641,662 bytes:
> Ok, indeed, from
> http://hadoop.apache.org/zookeeper/docs/r3.0.1/zookeeperAdmin.html#sc_configuration
> I see:
> jute.maxbuffer:
>     (Java system property:* jute.maxbuffer*)
>     This option can only be set as a Java system property. There is no
>     zookeeper prefix on it. It specifies the maximum size of the data
>     that can be stored in a znode. The default is 0xfffff, or just under
>     1M. If this option is changed, the system property must be set on
>     all servers and clients otherwise problems will arise. This is
>     really a sanity check. ZooKeeper is designed to store data on the
>     order of kilobytes in size.
> A more helpful exception would be nice :)
> Anybody have any experience popping this up a bit bigger?  What kind of
> bad things happen?
> Thanks,
> Eric

View raw message