Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@stratos.apache.org
Received-SPF: pass (nike.apache.org: domain of lasantha.fdo@gmail.com
 designates 209.85.216.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGJeazTjE-SzWjeKsw2G+-45+KqgtR9ggYz1OrgpAkq2e42pGA@mail.gmail.com>
References: 
 <CAGJeazTjE-SzWjeKsw2G+-45+KqgtR9ggYz1OrgpAkq2e42pGA@mail.gmail.com>
Date: Tue, 11 Nov 2014 20:51:41 +0530
Message-ID: 
 <CADWFnAp=2W+B=pMvFnU+2Kki06FvD7h2yQ-9ADwgWySn60uJgQ@mail.gmail.com>
Subject: Re: [Autoscaling] [Improvement] Introducing "curve fitting" for stat
 prediction algorithm of Autoscaler
From: Lasantha Fernando <lasantha.fdo@gmail.com>
To: dev@stratos.apache.org
Content-Type: text/plain; charset=UTF-8

Hi Lahiru,

Would it be possible to use linear regression already available as
Siddhi extensions in [1] or maybe improve on that existing extensions
to extend it to fit polynomial curves? The code is available here [2].

I think forecasting is also available which can be useful in this
usecase. WDYT? Just sharing my 2 cents.. :-)

[1] http://mail.wso2.org/mailarchive/architecture/2014-March/015696.html
[2] https://github.com/wso2-dev/siddhi/tree/master/modules/siddhi-extensions

Thanks,
Lasantha

On Tue, Nov 11, 2014 at 3:58 PM, Lahiru Sandaruwan <lahirus@wso2.com> wrote:
> Hi all,
>
> This contains the content i already sent to Stratos dev. Idea is to
> highlight and separate the new improvement.
>
> Current implementation
>
> Currently CEP calculates average, gradient, and second derivative and send
> those values to Autoscaler. Then Autoscaler predicts the values using S =
> u*t + 0.5*a*t*t.
>
> In this method CEP calculation is not very much accurate as it does not
> consider all the events when calculating the gradient and second derivative.
> Therefore the equation we apply doesn't yield the best prediction.
>
> Proposed Implementation
>
> CEP's task
>
> I think best approach is to do "curve fitting"[1] for received event sample
> in a particular time window. Refer "Locally weighted linear regression"
> section at [2] for more details.
>
> We would need a second degree polynomial fitter for this, where we can use
> Apache commons math library for this. Refer the sample at [3], we can run
> this with any degree. e.g. 2, 3. Just increase the degree to increase the
> accuracy.
>
> E.g.
> So if get degree 2 polynomial fitter, we will have an equation like below
> where value(v) is our statistic value and time(t) is the time of event.
>
> Equation we get from received events,
> v = a*t*t + b*t + c
>
> So the solution is,
>
> Find memberwise curves that fits events received in specific window(say 10
> minutes) at CEP
> Send the parameters of fitted line(a, b, and c in above equation) with the
> timestamp of last event(T) in the window, to Autoscaler
>
> Autoscaler's task
>
> Autoscaler use v = a*t*t + b*t + c function to predict the value in any
> timestamp from the last timestamp
>
> E.g. Say we need to find the value(v) after 1 minute(assuming we carried all
> the calculations in milliseconds),
>
> v = a * (T+60000) * (T+60000) + b * (T+60000) + c
>
> So we have memberwise predictions and we can find clusterwise prediction by
> averaging all the memberwise values.
>
>
> Please send your thoughts.
>
> Thanks.
>
> [1] http://en.wikipedia.org/wiki/Curve_fitting
> [2] http://cs229.stanford.edu/notes/cs229-notes1.pdf
> [3] http://commons.apache.org/proper/commons-math/userguide/fitting.html
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>