Mailing-List: contact user-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mahout.apache.org
Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.223.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1375907162.34575.YahooMailNeo@web126003.mail.ne1.yahoo.com>
References: 
 <CANNBgPKvAoBkEuYV76GxUpnEykFt45pVxLzkikmneAbWLwwnVA@mail.gmail.com>
 <CAJwFCa1D3kVYNovxYQLYWTzu93-qsHd-bpRZRP91JYhpvqDNjw@mail.gmail.com>
 <1375907162.34575.YahooMailNeo@web126003.mail.ne1.yahoo.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Wed, 7 Aug 2013 14:48:20 -0700
Message-ID: 
 <CAJwFCa13WMf1Zw8R55eVq=U=U2mahi13+VDi8JLTctBRMuxi2w@mail.gmail.com>
Subject: Re: Is OnlineSummarizer mergeable?
To: "user@mahout.apache.org" <user@mahout.apache.org>,
 Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Content-Type: multipart/alternative; boundary=047d7b10cf17c280cb04e36281fe

--047d7b10cf17c280cb04e36281fe
Content-Type: text/plain; charset=UTF-8

Otis,

What statistics do you need?

What guarantees?


On Wed, Aug 7, 2013 at 1:26 PM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> wrote:

> Hi Ted,
>
> I'm actually trying to find an alternative to QDigest (the stream-lib impl
> specifically) because even though it seems good, we have to deal with crazy
> volumes of data in SPM (performance monitoring service, see signature)...
> I'm hoping we can find something that has both a lower memory footprint
> than QDigest AND that is mergeable a la QDigest.  Utopia?
>
> Thanks,
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / Hadoop / HBase -
> http://sematext.com/spm
>
>
>
>
> >________________________________
> > From: Ted Dunning <ted.dunning@gmail.com>
> >To: "user@mahout.apache.org" <user@mahout.apache.org>
> >Sent: Wednesday, August 7, 2013 4:51 PM
> >Subject: Re: Is OnlineSummarizer mergeable?
> >
> >
> >It isn't as mergeable as I would like.  If you have randomized record
> >selection, it should be possible, but perverse ordering can cause serious
> >errors.
> >
> >It would be better to use something like a Q-digest.
> >
> >http://www.cs.virginia.edu/~son/cs851/papers/ucsb.sensys04.pdf
> >
> >
> >
> >
> >On Wed, Aug 7, 2013 at 4:21 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com
> >> wrote:
> >
> >> Hi,
> >>
> >> Is OnlineSummarizer algo "mergeable"?
> >>
> >> Say that we compute a percentile for some metric for time 12:00-12:01
> >> and store that somewhere, then we compute it for 1201-12:02 and store
> >> that separately, and so on.
> >>
> >> Can we then later merge these computed and previously stored
> >> percentile "instances" and get an accurate value?
> >>
> >> Thanks,
> >> Otis
> >> --
> >> Performance Monitoring -- http://sematext.com/spm
> >> Solr & ElasticSearch Support -- http://sematext.com/
> >>
> >
> >
> >

--047d7b10cf17c280cb04e36281fe--