Return-Path: Delivered-To: apmail-hadoop-zookeeper-user-archive@minotaur.apache.org Received: (qmail 6762 invoked from network); 13 May 2010 05:10:07 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 13 May 2010 05:10:07 -0000 Received: (qmail 83067 invoked by uid 500); 13 May 2010 05:10:07 -0000 Delivered-To: apmail-hadoop-zookeeper-user-archive@hadoop.apache.org Received: (qmail 82981 invoked by uid 500); 13 May 2010 05:10:06 -0000 Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: zookeeper-user@hadoop.apache.org Delivered-To: mailing list zookeeper-user@hadoop.apache.org Received: (qmail 82973 invoked by uid 99); 13 May 2010 05:10:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 May 2010 05:10:05 +0000 X-ASF-Spam-Status: No, hits=3.2 required=10.0 tests=AWL,FORGED_YAHOO_RCVD,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dirtyvagabond@gmail.com designates 74.125.83.48 as permitted sender) Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 May 2010 05:09:58 +0000 Received: by gwj20 with SMTP id 20so555793gwj.35 for ; Wed, 12 May 2010 22:09:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=dxrGTIcArZAdUrm9l8iOk23DZTsjn45w6W5tBn+9sfI=; b=qZEbbmyirKP8UkUS1euqsm/uiMC0Aq1yFp99E+0RuixmzI5pzZBBJ4brrFTHIT9sow +M7XYdjnE66JY0K1BO4NL9eB8wYyMyzOe4ueeujs9txJ+HeM3T2Dqi3J0ch4NyaohlMv LLnhj+tQ56POaidITxffVq14YuQupXYx7jHO4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; b=lJbqvQmVsDIujNr2I9/eop5P0ZhL1DNxVnr6uMsG5pR/M4tRDc58ZPZLaxXZXWuMXv 82mregDPo4g1nQ38qqUeRQ34A2KJhFn02ijOWYI+5UdFbtMjIMZ2dxjsrIjjPEw1yd2W MvUvSuJWsT4E117TJXCk+C/DFoMpevKOPSkeI= MIME-Version: 1.0 Received: by 10.101.129.10 with SMTP id g10mr5762933ann.31.1273727377599; Wed, 12 May 2010 22:09:37 -0700 (PDT) Sender: dirtyvagabond@gmail.com Received: by 10.101.69.15 with HTTP; Wed, 12 May 2010 22:09:37 -0700 (PDT) In-Reply-To: References: <4BD91D78.30909@apache.org> Date: Wed, 12 May 2010 22:09:37 -0700 X-Google-Sender-Auth: 8iP_OfMihaG9ddfJfD3yOgXv-3c Message-ID: Subject: Re: Pathological ZK cluster: 1 server verbosely WARN'ing, other 2 servers pegging CPU From: Aaron Crow To: zookeeper-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016368e28ffe1edab048672c399 --0016368e28ffe1edab048672c399 Content-Type: text/plain; charset=ISO-8859-1 Hi Ted, yeah it's a big number, eh? We're essentially using Zookeeper to track the state of cache entries, and currently we don't bound our cache. I didn't realize how many entries we grow to over a long period of time, until I started counting nodes in Zookeeper. But, sorry, I'm not sure what you mean about synchronized GC? You mean maybe all our Zookeepers get hit by a big GC at the same time, due to the very large db? On Wed, May 12, 2010 at 9:21 PM, Ted Dunning wrote: > Impressive number here, especially at your quoted "few per second" rate. > > Are you sure that you haven't inadvertently synchronized GC on multiple > machines? > > On Wed, May 12, 2010 at 8:30 PM, Aaron Crow > wrote: > > > Right now we're at > > 1.9 million. This isn't a bug of our application; it's actually a feature > > (but perhaps an ill-conceived one). > > > --0016368e28ffe1edab048672c399--