Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.198.239 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=qlX4ApD65m3KdHo3PUiPPnCmV7CCjwbzKHbSz4PJIw+dOa+MsO4MXBsS574YVd65al
         AwFMzocX/R6QT4DgzcjvzE+Vm0PDVz0jnnu4QihJrGyRa3U6fDTNQx6NEB02H52TPD1K
         tCD/UlVPK3FWbqyDlhA0ep70tOsJU1Aobr+0A=
MIME-Version: 1.0
In-Reply-To: <4E1BB3D8-950D-4851-A87C-9A64FA0934AD@gmail.com>
References: <BDF4F73D-77B1-4805-9A95-2267134A7618@apache.org>
	 <e2111bbb0901291526x710c5edamddfe7130c70d8224@mail.gmail.com>
	 <D74A64FE-3349-4694-A456-01D1884529C6@gmail.com>
	 <e2111bbb0901292157o33eac888k78a984dc7e6be51@mail.gmail.com>
	 <4E1BB3D8-950D-4851-A87C-9A64FA0934AD@gmail.com>
Date: Fri, 30 Jan 2009 02:02:36 -0500
Message-ID: <e2111bbb0901292302x74a6d6do6426368db03c5482@mail.gmail.com>
Subject: Re: Statistics Module
From: Paul Davis <paul.joseph.davis@gmail.com>
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Fri, Jan 30, 2009 at 1:58 AM, Antony Blakey <antony.blakey@gmail.com> wrote:
>
> On 30/01/2009, at 4:27 PM, Paul Davis wrote:
>
>> On Fri, Jan 30, 2009 at 12:32 AM, Antony Blakey <antony.blakey@gmail.com>
>> wrote:
>>>
>>> On 30/01/2009, at 9:56 AM, Paul Davis wrote:
>>>
>>>> The way that stats are calculated currently with the dependent
>>>> variable being time could cause some issues in implementing more
>>>> statistics. With my extremely limited knowledge of stats I think
>>>> moving that to be dependent on the number of requests might be better.
>>>> This is something that hopefully someone out there knows more about.
>>>> (This is in terms of "avg for last 5 minutes" vs "avg for last 100
>>>> requests", (the later of the two making stddev type stats
>>>> calculateable on the fly in constant memory.)
>>>
>>> The problem with using # of requests is that depending on your data, each
>>> request may take a long time. I have this problem at the moment: 1008
>>> documents in a 3.5G media database. During a compact, the status in
>>> _active_tasks updates every 1000 documents, so you can imagine how useful
>>> that is :/ I thought it had hung (and neither the beam.smp CPU time nor
>>> the
>>> IO requests were a good indicator). I spent some time chasing this down
>>> as a
>>> bug before realising the problems was in the status granularity!
>>>
>>
>> Actually I don't think that affects my question at all. It may change
>> how we report things though. As in, it may be important to be able to
>> report things that are not single increment/decrement conditions but
>> instead allow for adding arbitrary floating point numbers to the
>> number of recorded data points.
>
> I think I have the wrong end of the stick here - my problem was with the
> granularity of updates, not with the basis of calculation.
>

Heh. Well, we can only measure what we know. And in the interest of
simplicity I think the granularity is gonna have to stick to pretty
much per request. Also you're flying with 300 MiB docs? perhaps its
time to chop or store in FTP?

> Antony Blakey
> -------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> It is no measure of health to be well adjusted to a profoundly sick society.
>  -- Jiddu Krishnamurti
>
>
>