Mailing-List: contact dev-help@systemml.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@systemml.apache.org
MIME-Version: 1.0
In-Reply-To: <CAObMueWa_J-DcbWv-nzkeYxWNS+VbuirFfRcSv1U3Veb=e7azw@mail.gmail.com>
References: <CAObMueWa_J-DcbWv-nzkeYxWNS+VbuirFfRcSv1U3Veb=e7azw@mail.gmail.com>
From: Janardhan Pulivarthi <janardhan.pulivarthi@gmail.com>
Date: Tue, 25 Jul 2017 23:02:46 +0530
Message-ID: <CAObMueVmk5DXnuALQp_WX2b01F6YkmHbp_sorbhcyphx0p7WYA@mail.gmail.com>
Subject: Re: Bayesian optimizer support for SystemML.
To: dev@systemml.apache.org
Content-Type: multipart/alternative; boundary="001a11449fba9de532055527ba3b"
archived-at: Tue, 25 Jul 2017 17:33:45 -0000

--001a11449fba9de532055527ba3b
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Niketan and Mike,

As we are trying to implement this Bayesian Optimization, should we take
input from more committers as well as this optimizer approach seems to have
a couple of ways to implement. We may need to find out which suits us the
best.

Thanks,
Janardhan

On Sat, Jul 22, 2017 at 3:41 PM, Janardhan Pulivarthi <
janardhan.pulivarthi@gmail.com> wrote:

> Dear committers,
>
> We will be planning to add bayesian optimizer support for both the ML and
> Deep learning tasks for the SystemML. Relevant jira link:
> https://issues.apache.org/jira/browse/SYSTEMML-979
>
> The following is a simple outline of how we are going implement it. Pleas=
e
> feel free to make any kind of changes. In this google docs link:
> http://bit.do/systemml-bayesian
>
> Description:
>
> Bayesian optimization is a sequential design strategy for global
> optimization of black-box functions that doesn=E2=80=99t require derivati=
ves.
>
> Process:
>
>    1.
>
>    First we select a point that will be the best as far as the no. of
>    iterations that has happened.
>    2.
>
>    Candidate point selection with sampling from Sobol quasirandom
>    sequence generator the space.
>    3.
>
>    Gaussian process hyperparameter sampling with surrogate slice sampling
>    method.
>
>
> Components:
>
>    1.
>
>    Selecting the next point to Evaluate.
>
> [image: nextpoint.PNG]
>
> We specify a uniform prior for the mean, m, and width 2 top-hat priors fo=
r
> each of the D length scale parameters. As we expect the observation noise
> generally to be close to or exactly zero, v(nu) is given a horseshoe
> prior. The covariance amplitude theta0 is given a zero mean, unit varianc=
e
> lognormal prior, theta0 ~ ln N (0, 1).
>
>
>
>    1.
>
>    Generation of QuasiRandom Sobol Sequence.
>
> Which kind of sobol patterns are needed?
>
> [image: sobol patterns.PNG]
>
> How many dimensions do we need?
>
> This paper argues that its generation target dimension is 21201. [pdf lin=
k
> <https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe%20c=
onstructing.pdf>
> ]
>
>
>
>    1.
>
>    Surrogate Slice Sampling.
>
> [image: surrogate data sampling.PNG]
>
>
> References:
>
> 1. For the next point to evaluate:
>
> https://papers.nips.cc/paper/4522-practical-bayesian-
> optimization-of-machine-learning-algorithms.pdf
>
>  http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips_appendix.pdf
>
>
> 2. QuasiRandom Sobol Sequence Generator:
>
> https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe%
> 20constructing.pdf
>
>
> 3. Surrogate Slice Sampling:
>
> http://homepages.inf.ed.ac.uk/imurray2/pub/10hypers/hypers.pdf
>
>
>
> Thank you so much,
>
> Janardhan
>
>
>
>

--001a11449fba9de532055527ba3b--