commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Berlin <>
Subject Re: [Math] Fluent API, inheritance and immutability
Date Thu, 15 Aug 2013 13:12:58 GMT

As a quick first pass, I will comment on Evans code only for now. Most
of my comments have to do with numerical aspects and associated
design, rather than threading.

I like the idea of the optimize function taking in a Problem class. I
think it can add a lot of flexibility going forward.

First I think to make things clear, LeastSquares should be renamed to
IterativeLeastSquares. Since for unconstrained linear least squares,
this whole hierarchy is not correct. For one thing they don't need an
initial guess, for another, linear least squares can factor the matrix
before experimental data is supplied, so it can support rapid

There are several issues that I have with current example

1) While the least-squares problem now encapsulates the weights, which
I like, for some reason the weights are explicitly used in the actual
GaussNewtonOptimizer and probably all other least-squares optimizer. I
think that is bad for several reasons:
i) the actual optimizer algorithm has nothing to do with weights, so
it logically inconsistent. It forces people who have no weights to
still go through the actual numerical computation.
ii) it makes it harder to maintain and update the optimizer code. Just
think about how much cleaner it would be if weights are actually
included in the Jacobian.
iIi) The inclusion of weight can be really optimized inside the
LeastSquaresProblem, where depending on the problem they might be able
to take it more efficiently than the general approach.

2) All computations inside the optimizers should be done using linear
algebra library (or is the library so bad?). Not only does it make the
code easier to follow, it can make it a lot faster. Imagine a
not-so-hypothetical example where the Jacobian is actually sparse
(I know you guys are removing sparse linear algebra, but lets say it
gets put back at some point later). Forming the Hessian from the
Jacobian is a very expensive operation (J^T*J), but if J is sparse the
speedup could be automatically propagated through the algorithm.

3) Speaking of forming the Hessian, the actual computation of the
descent step should be logically and explicitly separated into two
separate function. I think the first function could actually be moved
to the LeastSquaresProblem (need more thought on this)
 i) The first function will compute the step size p, where H*p = -g, H
is the Hessian and g is the gradient. The reason for this is there are
several ways this problem can be solved, and that should be deferred
to a specific implementation later. For example, in a very large
least-squares problem (or any large problem) you cannot actually form
J^T*J (or H), and instead you solve the problem using conjugate
gradient iterative method (which you already have in the library),
where you compute J^T*J*x as J^T*(J*x). Again, here having the
Jacobian be a matrix would really help future proof things. In the
case of general non-linear optimizers you might want to support in the
future a very important quasi-Newton algorithms BFGS or L-BFGS, were
this step is done very differently.

ii) Once the step size is computed, a line search or trust region
method is used to potentially modify the step. Here also bound
constraints can be enforced, so an optimizer that supports it can
overwrite this function. This part should also be exposed, at least in
the abstract classes, from which the final least-squares optimizer
should be built.


On Thu, Aug 15, 2013 at 8:31 AM, Gilles <> wrote:
> On Wed, 14 Aug 2013 23:40:58 +0100, sebb wrote:
>> On 14 August 2013 23:34, Gilles <> wrote:
>>> At this point, I'd tend to think that creating a copy of trunk in the
>>> Commons's "sandbox" part of the repository will be more productive.
>> Why sandbox? Just use a temporary branch in the existing math repo.
> Has Evan the permissions to commit there?
>>> There, we can both directly modify the code to make a point and
>>> converge to a design while incrementally ensure that every features
>>> turn out as imagined.
> Gilles
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message