commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gilles <>
Subject Re: [Math] LeastSquaresOptimizer Design
Date Tue, 22 Sep 2015 11:46:32 GMT

On Mon, 21 Sep 2015 19:55:15 -0500, Ole Ersoy wrote:
> Hola,
> On 09/21/2015 04:15 PM, Gilles wrote:
>> Hi.
>> On Sun, 20 Sep 2015 15:04:08 -0500, Ole Ersoy wrote:
>>> On 09/20/2015 05:51 AM, Gilles wrote:
>>>> On Sun, 20 Sep 2015 01:12:49 -0500, Ole Ersoy wrote:
>>>>> Wanted to float some ideas for the LeastSquaresOptimizer 
>>>>> (Possibly
>>>>> General Optimizer) design.  For example with the
>>>>> LevenbergMarquardtOptimizer we would do:
>>>>> `LevenbergMarquardtOptimizer.optimize(OptimizationContext c);`
>>>>> Rough optimize() outline:
>>>>> public static void optimise() {
>>>>> //perform the optimization
>>>>> //If successful
>>>>>     c.notify(LevenberMarquardtResultsEnum.SUCCESS, solution);
>>>>> //If not successful
>>>>> c.notify(LevenberMarquardtResultsEnum.TOO_SMALL_COST_RELATIVE_TOLERANCE,
>>>>> diagnostic);
>>>>> //or
>>>>> c.notify(LevenberMarquardtResultsEnum.TOO_SMALL_PARAMETERS_RELATIVE_TOLERANCE,
>>>>> diagnostic)
>>>>> //etc
>>>>> }
>>>>> The diagnostic, when turned on, will contain a trace of the last 
>>>>> N
>>>>> iterations leading up to the failure.  When turned off, the 
>>>>> Diagnostic
>>>>> instance only contains the parameters used to detect failure. The
>>>>> diagnostic could be viewed as an indirect way to log optimizer
>>>>> iterations.
>>>>> WDYT?
>>>> I'm wary of having several different ways to convey information to 
>>>> the
>>>> caller.
>>> It would just be one way.
>> One way for optimizer, one way for solvers, one way for ...
> Yes I see what you mean, but I think on a whole it will be worth it
> to add additional sugar code that removes the need for exceptions.

Isn't always possible to wrap exception-generating code so that upper
layers do not see them?
The interface would have to know how to handle them and propagate the
information in some other form (callback).

>>> But the caller may not be the receiver
>>> (It could be).  The receiver would be an observer attached to the
>>> OptimizationContext that implements an interface allowing it to 
>>> observe
>>> the optimization.
>> I'm afraid that it will add to the questions of what to put in the
>> code and how.  [We already had sometimes heated discussions just for
>> the IMHO obvious (e.g. code formatting, documentation, 
>> exception...).]
> Hehe.  Yes I remember some of these discussions.  I wonder how much
> time was spent debating the exceptions alone?  Surely everyone must
> have had this feeling in pit of their stomach that there's got to be 
> a
> better way.  On the exception topic, these are some of the issues:
> I18N
> ===================
> If you are new to commons math and thinking about designing a commons
> math compatible exception you should probably understand the I18N
> stuff that's bound to exception (and wonder why it's bound the the
> exception).  Grab a coffee and spend a few hours, unless you are
> obviously fairly new to Java like some ofthe people posting for help.
> In this case when the exception occurs, there is going to be a lot of
> tutoring going on on the users list.

I already said all I had to say about this; it's in the archive.
Summary: I agree that it shouldn't be here.

> Number of Exceptions
> ===================
> Before you do actually design a new exception, you should probably
> see if there is an exception that already fits the category of what
> you are doing.  So you start reading.  Exception1...nop
> Exception2...nop...Exception3...Exception999..But I think I'm getting
> warmer.  OK - Did not find it ... but I'm fairly certain that there 
> is
> a elegant place for it somewhere in the exception hierarchy...

On this, I also explained at length my views (assuming that exceptions
are part of the design).
Summary: an exception indicates that something went wrong, and the 
should not hope to get anything good out of the call that raised the
exception (i.e. he _must_ craft another call that meets the 
of the code).

> Handling of Exceptions
> ===================
> If our app uses several of the commons math classes (That throw
> exceptions of the same type), and one of those classes throws an
> exception,what is the app supposed to do?

Cf. previous paragraph.

> I think most developers would find that question somewhat
> challenging.  There are numerous strategies.  Catch all exceptions 
> and
> log what happened, etc.  But what if the requirement is that if an
> exception is thrown, the organization that receives it has 0 seconds
> to get to the root cause of it and understand the dynamics. Is this
> doable?  (Yes obviously, but how hard is it...?).

Cf. previous paragraph.
In effect, you describe an upper layer's requirement (handling an 
"unexpected(!) failure"). IMHO, it's out CM's realm (CM raises the 
end of story).

>>>> It seems that the reporting interfaces could quickly overwhelm
>>>> the "actual" code (one type of context per algorithm).
>>> There would one type of Observer interface per algorithm.  It would
>>> act on the solution and what are currently exceptions, although 
>>> these
>>> would be translated into enums.
>> Unless I'm mistaken, the most common use-case for codes implemented
>> in a library such as CM is to provide a correct answer or bail out
>> in a non-equivocal way.
> Most java developers are used to synchronous the method
> get the response...catch the exception if needed.  This is changing
> with JDK8, and as we evolve and start using lambdas, we become more
> accustomed to the functional callback style of programming.
> Personally I want to be able to use an API that gives me what I need
> when everything works as expected, allows me to resolve unexpected
> issues with minimal effort, and is as simple, fluid, and lightweight
> as possible.

I've not yet used Java 8; I would have if we were allowed to use it in

However, I'm not convinced that asynchronicity should be dealt with
at the CM level, beyond making its algorithms multi-thread friendly.
IMO, this is the important change (that can make a big difference,
performance-wise, on machines with multiple cores).
Then developers can use the standard tools in "java.util.concurrent"
to select a runtime policy (single/multi-thread and/or (a)synchronous).

>> It would make the code more involved to handle a minority of
>> (undefined) cases. [Actual examples would be welcome in order to
>> focus the discussion.]
> Rough Outline (I've evolved the concept and moved away from the
> OptimizationContext in the process of writing):
> interface LevenbergMarquardtObserver {
>     public void hola(Solution s);
>     public void sugarHoneyIceTea(ResultType rt, Dianostics d);
> }
> public class LMObserver implements LevenbergMarquardtObserver {
>    private Application application;
>    public LMObserver(Application application) {
>        this.application = application;
>    }
>    public void hola(ResultType rt, Solution s) {
>       ;
>    }
>    public void sugarHoneyIceTea(ResultType rt, Diagnostic s)
>        if (rt == ResultType.I_GOT_THIS_ONE) {
>             //I looked at the commons unit tests for this algorithm
> evaluating
>             //the diagnostics that shows how this failure can occur
>             //I'm totally fixing this!  Steps aside!
>        }
>        else if (rt == ResultType.REALLY_COMPLICATED_STUFF)
>        {
>            //We need our best India.
>        }
>   )
> public class Application {
>     //Note nothing is returned.
>     LevenberMarquardtOptimizer.setOberver(new
> LMObserver(this)).setLeastSquaresProblem(new
> ClassThatImplementsTheProblem())).start();
>     public void next(Solution solution) {
>         //Do cool stuff.
>     }
> }
> Or an asynchronous variation:
> public class Application {
> //This call will not block because async is true
>     LevenberMarquardtOptimizer.setAsync(true).setOberver(new
> LMObserver()).setLeastSquaresProblem(new
> ClassThatImplementsTheProblem())).start();
>     //Do more stuff right away.
>     public void next(Solution solution) {
>         //When the thread running the optimization is done, this
> method is called back.
>         //Do whatever comes next
>     }
> }
> The above would start the optimization in a separate thread that does
> not / SHOULD NOT share data with the main thread.

Cf. previous paragraph: I think that can be done in a layer above CM.

>>>> The current reporting is based on exceptions, and assumes that if 
>>>> no
>>>> exception was thrown, then the user's request completed 
>>>> successfully.
>>> Sure - personally I'd much rather deal with something similar to an
>>> HTTP status code in a callback, than an exception .  I think the 
>>> code
>>> is cleaner and the calback makes it more elegant to apply an 
>>> adaptive
>>> approach to handling the response, like slightly relaxing 
>>> constraints,
>>> convergence parameters, etc.  Also by getting rid of the 
>>> exceptions,
>>> we no longer depend on the I18N layer that they are tied to and now
>>> the messages can be more informative, since they target the root
>>> cause.  The observer can also run in the 'main' thread' while the
>>> optimization can run asynchronously.  Also WRT JDK9 and modules,
>>> loosing the exceptions would mean one less dependency when the 
>>> library
>>> is up into JDK9 modules...which would be more in line with this
>>> philosophy:
>> I'm not sure I fully understood the philosophy from the text in this
>> short paragraph.
>> But I do not agree with the idea that the possibility to quickly 
>> find
>> some code is more important than standards and best practices.
> If you go to and type in Neural Network you will get 56
> results all linked to github repositories.
> In addition there's meta data indicating number of downloads in the
> last day, last month, etc.  Try typing in cosine.  Odds are you will
> find a package that does just want you want and nothing else.  This 
> is
> very underwhelming and refreshing in terms of cloning off of github
> and getting familar with tests etc.  Also eye opening.  How many of 
> us
> knew that we could do that much stuff with cosine! :).

I really don't mean to question the quality of any of those 
but the issue is there: How to choose?
That there are so many of them sort of defeats the purpose of "quickly
find what you need".

It seems (?) that the consequence of this modularity (?) is to 
the creation of many independent/competing/duplicate projects of small
teams (I'd guess, a 1-person-team, in most cases).

>>>> I totally agree that in some circumstances, more information on 
>>>> the
>>>> inner working of an algorithm would be quite useful.
>>> ... Algorithm iterations become unit testable.
>>>> But I don't see the point in devoting resources to reinvent the 
>>>> wheel:
>>> You mean pimping the wheel?  Big pimpin.
>> I think that logging statements are easy to add, not disruptive at 
>> all,
>> and come in handy to understand a code's unexpected behaviour.
>> Assuming that a "logging" feature is useful, it can be added *now* 
>> using
>> a dependency towards a weight-less (!) framework such as "slf4j".
>> IMO, it would be a waste of time to implement a new communication 
>> layer
>> that can do that, and more, if it would be used for logging only in 
>> 99%
>> of the cases.
> SLF4J is used by almost every other framework, so why not use it?

Good question: I also asked it quite some time ago.
Didn't get a satisfying answer. Boiled down to "no dependency" policy.

> Logging and the diagnostic could be used together.  The primary
> purpose of the diagnostic though is to collect data that will be
> useful in `sugarHoneyIceTea`.

I'm not sure I understand correctly the purpose: if the "Solution" is
found, do you ever need more "context" (i.e. "Result", "Diagnostics")?

If it is only necessary in case of failure, CM's exception can already
carry context information.  As I wrote above, such an exception could
be caught by a wrapper (not necessarily part of the CM "core") and
translated into whatever the upper layer expect (e.g. "Diagnostics").

>>>> I longed several times for the use of a logging library.
>>>> The only show-stopper has been the informal "no-dependency" 
>>>> policy...
>>> JDK9 Jigsaw should solve dependency hell, so the less coupling
>>> between commons math classes the better.
>> I wouldn't call "coupling" the dependency towards exception classes:
>> they are little utilities that can make sense in various parts of 
>> the
>> library.
> If for example the Simplex solver is broken off into it's own module,
> then it has to be coupled to the exceptions, unless it is exception
> free.

Why is it a problem to be coupled with a few tiny exception classes?

Then if it is really a problem, we can indeed define "local" exceptions
for each package.

>> [Unless one wants to embark on yet another discussion about 
>> exceptions;
>> whether there should be one class for each of the "messages" that 
>> exist
>> in "LocalizedFormats"; whether localization should be done in CM;
>> etc.]
> I think it would be best to just eliminate the exceptions.

I'd think that most users of CM should deem that dangerous.
An exception is relatively difficult to ignore unknowingly (and was
rightfully a better alternative the old "check the return value").

>>> Anyways I'm obviously
>>> interested in playing with this stuff, so when I get something up 
>>> into
>>> a repository I'll to do a callback :).
>> If you are interested in big overhauls, there is one that gathered
>> relative consensus: rewrite the algorithms in a 
>> "multithread-friendly"
>> way.
> I think that's a tall order that will take us into JDK88 :).

That would be a real pity.
I recall a nit-picking discussion about how to initialize the 
class in order to gain a few _milliseconds_. :-/

> But
> using callbacks and making potentially long running computations
> asynchronous could be a middle ground that would allow simple multi
> threaded use without fiddling around under the hood...

Cf. above (this does not need ad-hoc CM code, beyond the relevant 
implementing "Runnable" and/or "Callable").

>> Some ideas were floated (cf. ML archive) but no implementation or
>> experiment...  Perhaps with a well-defined goal such as performance
>> improvement, your design suggestions will become clearer to more 
>> people.
>> AFAIK, only the classes in the "o.a.c.m.neuralnet" package are 
>> currently
>> ready to be used with the "java.util.concurrent" framework.
> FWIU Neural Nets are a great fit for concurrency.

Quite true.

But even the optimizers could benefit from just being able to use
more threads: It is often (always?) necessary to evaluate the objective
function "N" times per iteration. So, the computation could be about
   min(N, numCores)
times faster.

> I think for the
> others we will end up having discussions around how users would
> control the number of threads, etc. again that makes some of us
> nervous.

One additional parameters: numCores.

> An asynchronous operation that runs in one separate thread
> is easier to reason about.


But then we should stop talking about performance on this list. ;-}

> If we want to test 10 neural net
> configurations, and we have 10 cores, then we can start each by 
> itself
> by doing something like:
> Nework.setAsync(true).addNeurons().connectNeurons().addObserver(observer).start().
> //Now do 10 more
> //If the observer is shared then notifications should be thread safe.

I had a similar argument for not making "FastMath" initialization 
(at the cost of a lot of additional code):  It was rejected...


P.S. I think that several issues evoked in this thread could warrant 
      their own thread, to gather more opinions on actual actions to be 

> Cheers,
> - Ole
> P.S. Dang that was a long email.  If I write one more of these, ban 
> me :)

My fault: I should not keep answering! ;-)

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message