commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ole Ersoy <>
Subject Re: [Math] LeastSquaresOptimizer Design
Date Tue, 22 Sep 2015 16:35:44 GMT
One more thing - This is separate from the other stuff.  The LMOptimizer has several configuration
properties, with corresponding getters and a corresponding with() API.  It would be good if
these existed on their own class that used Lombok ( to generate
(Byte code) the fluid API and the getters.  That would eliminate and estimated 40% of the
source code.  Lombok annotations, such as @NonNull, could be used to generate NPE checks in
the configuration instance.  This way CM does not have to check for NPEs anymore.  The will
be thrown when the configuration instance is constructed.

This is one reason I was thinking it might be useful to have an context holding things like:
- configuration
- callback
- problem

But I think it's fine if these are passed in separately as well. It's cleaner to call callback.notify()
than context.callback.notify().

- Ole

On 09/22/2015 06:46 AM, Gilles wrote:
> Hi.
> On Mon, 21 Sep 2015 19:55:15 -0500, Ole Ersoy wrote:
>> Hola,
>> On 09/21/2015 04:15 PM, Gilles wrote:
>>> Hi.
>>> On Sun, 20 Sep 2015 15:04:08 -0500, Ole Ersoy wrote:
>>>> On 09/20/2015 05:51 AM, Gilles wrote:
>>>>> On Sun, 20 Sep 2015 01:12:49 -0500, Ole Ersoy wrote:
>>>>>> Wanted to float some ideas for the LeastSquaresOptimizer (Possibly
>>>>>> General Optimizer) design.  For example with the
>>>>>> LevenbergMarquardtOptimizer we would do:
>>>>>> `LevenbergMarquardtOptimizer.optimize(OptimizationContext c);`
>>>>>> Rough optimize() outline:
>>>>>> public static void optimise() {
>>>>>> //perform the optimization
>>>>>> //If successful
>>>>>>     c.notify(LevenberMarquardtResultsEnum.SUCCESS, solution);
>>>>>> //If not successful
>>>>>> c.notify(LevenberMarquardtResultsEnum.TOO_SMALL_COST_RELATIVE_TOLERANCE,
>>>>>> diagnostic);
>>>>>> //or
>>>>>> c.notify(LevenberMarquardtResultsEnum.TOO_SMALL_PARAMETERS_RELATIVE_TOLERANCE,
>>>>>> diagnostic)
>>>>>> //etc
>>>>>> }
>>>>>> The diagnostic, when turned on, will contain a trace of the last
>>>>>> iterations leading up to the failure.  When turned off, the Diagnostic
>>>>>> instance only contains the parameters used to detect failure. The
>>>>>> diagnostic could be viewed as an indirect way to log optimizer
>>>>>> iterations.
>>>>>> WDYT?
>>>>> I'm wary of having several different ways to convey information to the
>>>>> caller.
>>>> It would just be one way.
>>> One way for optimizer, one way for solvers, one way for ...
>> Yes I see what you mean, but I think on a whole it will be worth it
>> to add additional sugar code that removes the need for exceptions.
> Isn't always possible to wrap exception-generating code so that upper
> layers do not see them?
> The interface would have to know how to handle them and propagate the
> information in some other form (callback).
>>>> But the caller may not be the receiver
>>>> (It could be).  The receiver would be an observer attached to the
>>>> OptimizationContext that implements an interface allowing it to observe
>>>> the optimization.
>>> I'm afraid that it will add to the questions of what to put in the
>>> code and how.  [We already had sometimes heated discussions just for
>>> the IMHO obvious (e.g. code formatting, documentation, exception...).]
>> Hehe.  Yes I remember some of these discussions.  I wonder how much
>> time was spent debating the exceptions alone?  Surely everyone must
>> have had this feeling in pit of their stomach that there's got to be a
>> better way.  On the exception topic, these are some of the issues:
>> I18N
>> ===================
>> If you are new to commons math and thinking about designing a commons
>> math compatible exception you should probably understand the I18N
>> stuff that's bound to exception (and wonder why it's bound the the
>> exception).  Grab a coffee and spend a few hours, unless you are
>> obviously fairly new to Java like some ofthe people posting for help.
>> In this case when the exception occurs, there is going to be a lot of
>> tutoring going on on the users list.
> I already said all I had to say about this; it's in the archive.
> Summary: I agree that it shouldn't be here.
>> Number of Exceptions
>> ===================
>> Before you do actually design a new exception, you should probably
>> see if there is an exception that already fits the category of what
>> you are doing.  So you start reading.  Exception1...nop
>> Exception2...nop...Exception3...Exception999..But I think I'm getting
>> warmer.  OK - Did not find it ... but I'm fairly certain that there is
>> a elegant place for it somewhere in the exception hierarchy...
> On this, I also explained at length my views (assuming that exceptions
> are part of the design).
> Summary: an exception indicates that something went wrong, and the caller
> should not hope to get anything good out of the call that raised the
> exception (i.e. he _must_ craft another call that meets the requirements
> of the code).
>> Handling of Exceptions
>> ===================
>> If our app uses several of the commons math classes (That throw
>> exceptions of the same type), and one of those classes throws an
>> exception,what is the app supposed to do?
> Cf. previous paragraph.
>> I think most developers would find that question somewhat
>> challenging.  There are numerous strategies.  Catch all exceptions and
>> log what happened, etc.  But what if the requirement is that if an
>> exception is thrown, the organization that receives it has 0 seconds
>> to get to the root cause of it and understand the dynamics. Is this
>> doable?  (Yes obviously, but how hard is it...?).
> Cf. previous paragraph.
> In effect, you describe an upper layer's requirement (handling an expected
> "unexpected(!) failure"). IMHO, it's out CM's realm (CM raises the exception,
> end of story).
>>>>> It seems that the reporting interfaces could quickly overwhelm
>>>>> the "actual" code (one type of context per algorithm).
>>>> There would one type of Observer interface per algorithm. It would
>>>> act on the solution and what are currently exceptions, although these
>>>> would be translated into enums.
>>> Unless I'm mistaken, the most common use-case for codes implemented
>>> in a library such as CM is to provide a correct answer or bail out
>>> in a non-equivocal way.
>> Most java developers are used to synchronous the method
>> get the response...catch the exception if needed.  This is changing
>> with JDK8, and as we evolve and start using lambdas, we become more
>> accustomed to the functional callback style of programming.
>> Personally I want to be able to use an API that gives me what I need
>> when everything works as expected, allows me to resolve unexpected
>> issues with minimal effort, and is as simple, fluid, and lightweight
>> as possible.
> I've not yet used Java 8; I would have if we were allowed to use it in
> CM...
> However, I'm not convinced that asynchronicity should be dealt with
> at the CM level, beyond making its algorithms multi-thread friendly.
> IMO, this is the important change (that can make a big difference,
> performance-wise, on machines with multiple cores).
> Then developers can use the standard tools in "java.util.concurrent"
> to select a runtime policy (single/multi-thread and/or (a)synchronous).
>>> It would make the code more involved to handle a minority of
>>> (undefined) cases. [Actual examples would be welcome in order to
>>> focus the discussion.]
>> Rough Outline (I've evolved the concept and moved away from the
>> OptimizationContext in the process of writing):
>> interface LevenbergMarquardtObserver {
>>     public void hola(Solution s);
>>     public void sugarHoneyIceTea(ResultType rt, Dianostics d);
>> }
>> public class LMObserver implements LevenbergMarquardtObserver {
>>    private Application application;
>>    public LMObserver(Application application) {
>>        this.application = application;
>>    }
>>    public void hola(ResultType rt, Solution s) {
>>       ;
>>    }
>>    public void sugarHoneyIceTea(ResultType rt, Diagnostic s)
>>        if (rt == ResultType.I_GOT_THIS_ONE) {
>>             //I looked at the commons unit tests for this algorithm
>> evaluating
>>             //the diagnostics that shows how this failure can occur
>>             //I'm totally fixing this!  Steps aside!
>>        }
>>        else if (rt == ResultType.REALLY_COMPLICATED_STUFF)
>>        {
>>            //We need our best India.
>>        }
>>   )
>> public class Application {
>>     //Note nothing is returned.
>>     LevenberMarquardtOptimizer.setOberver(new
>> LMObserver(this)).setLeastSquaresProblem(new
>> ClassThatImplementsTheProblem())).start();
>>     public void next(Solution solution) {
>>         //Do cool stuff.
>>     }
>> }
>> Or an asynchronous variation:
>> public class Application {
>> //This call will not block because async is true
>>     LevenberMarquardtOptimizer.setAsync(true).setOberver(new
>> LMObserver()).setLeastSquaresProblem(new
>> ClassThatImplementsTheProblem())).start();
>>     //Do more stuff right away.
>>     public void next(Solution solution) {
>>         //When the thread running the optimization is done, this
>> method is called back.
>>         //Do whatever comes next
>>     }
>> }
>> The above would start the optimization in a separate thread that does
>> not / SHOULD NOT share data with the main thread.
> Cf. previous paragraph: I think that can be done in a layer above CM.
>>>>> The current reporting is based on exceptions, and assumes that if no
>>>>> exception was thrown, then the user's request completed successfully.
>>>> Sure - personally I'd much rather deal with something similar to an
>>>> HTTP status code in a callback, than an exception .  I think the code
>>>> is cleaner and the calback makes it more elegant to apply an adaptive
>>>> approach to handling the response, like slightly relaxing constraints,
>>>> convergence parameters, etc.  Also by getting rid of the exceptions,
>>>> we no longer depend on the I18N layer that they are tied to and now
>>>> the messages can be more informative, since they target the root
>>>> cause.  The observer can also run in the 'main' thread' while the
>>>> optimization can run asynchronously.  Also WRT JDK9 and modules,
>>>> loosing the exceptions would mean one less dependency when the library
>>>> is up into JDK9 modules...which would be more in line with this
>>>> philosophy:
>>> I'm not sure I fully understood the philosophy from the text in this
>>> short paragraph.
>>> But I do not agree with the idea that the possibility to quickly find
>>> some code is more important than standards and best practices.
>> If you go to and type in Neural Network you will get 56
>> results all linked to github repositories.
>> In addition there's meta data indicating number of downloads in the
>> last day, last month, etc.  Try typing in cosine.  Odds are you will
>> find a package that does just want you want and nothing else. This is
>> very underwhelming and refreshing in terms of cloning off of github
>> and getting familar with tests etc.  Also eye opening.  How many of us
>> knew that we could do that much stuff with cosine! :).
> I really don't mean to question the quality of any of those implementations,
> but the issue is there: How to choose?
> That there are so many of them sort of defeats the purpose of "quickly
> find what you need".
> It seems (?) that the consequence of this modularity (?) is to encourage
> the creation of many independent/competing/duplicate projects of small
> teams (I'd guess, a 1-person-team, in most cases).
>>>>> I totally agree that in some circumstances, more information on the
>>>>> inner working of an algorithm would be quite useful.
>>>> ... Algorithm iterations become unit testable.
>>>>> But I don't see the point in devoting resources to reinvent the wheel:
>>>> You mean pimping the wheel?  Big pimpin.
>>> I think that logging statements are easy to add, not disruptive at all,
>>> and come in handy to understand a code's unexpected behaviour.
>>> Assuming that a "logging" feature is useful, it can be added *now* using
>>> a dependency towards a weight-less (!) framework such as "slf4j".
>>> IMO, it would be a waste of time to implement a new communication layer
>>> that can do that, and more, if it would be used for logging only in 99%
>>> of the cases.
>> SLF4J is used by almost every other framework, so why not use it?
> Good question: I also asked it quite some time ago.
> Didn't get a satisfying answer. Boiled down to "no dependency" policy.
>> Logging and the diagnostic could be used together.  The primary
>> purpose of the diagnostic though is to collect data that will be
>> useful in `sugarHoneyIceTea`.
> I'm not sure I understand correctly the purpose: if the "Solution" is
> found, do you ever need more "context" (i.e. "Result", "Diagnostics")?
> If it is only necessary in case of failure, CM's exception can already
> carry context information.  As I wrote above, such an exception could
> be caught by a wrapper (not necessarily part of the CM "core") and
> translated into whatever the upper layer expect (e.g. "Diagnostics").
>>>>> I longed several times for the use of a logging library.
>>>>> The only show-stopper has been the informal "no-dependency" policy...
>>>> JDK9 Jigsaw should solve dependency hell, so the less coupling
>>>> between commons math classes the better.
>>> I wouldn't call "coupling" the dependency towards exception classes:
>>> they are little utilities that can make sense in various parts of the
>>> library.
>> If for example the Simplex solver is broken off into it's own module,
>> then it has to be coupled to the exceptions, unless it is exception
>> free.
> Why is it a problem to be coupled with a few tiny exception classes?
> Then if it is really a problem, we can indeed define "local" exceptions
> for each package.
>>> [Unless one wants to embark on yet another discussion about exceptions;
>>> whether there should be one class for each of the "messages" that exist
>>> in "LocalizedFormats"; whether localization should be done in CM;
>>> etc.]
>> I think it would be best to just eliminate the exceptions.
> I'd think that most users of CM should deem that dangerous.
> An exception is relatively difficult to ignore unknowingly (and was
> rightfully a better alternative the old "check the return value").
>>>> Anyways I'm obviously
>>>> interested in playing with this stuff, so when I get something up into
>>>> a repository I'll to do a callback :).
>>> If you are interested in big overhauls, there is one that gathered
>>> relative consensus: rewrite the algorithms in a "multithread-friendly"
>>> way.
>> I think that's a tall order that will take us into JDK88 :).
> That would be a real pity.
> I recall a nit-picking discussion about how to initialize the "FastMath"
> class in order to gain a few _milliseconds_. :-/
>> But
>> using callbacks and making potentially long running computations
>> asynchronous could be a middle ground that would allow simple multi
>> threaded use without fiddling around under the hood...
> Cf. above (this does not need ad-hoc CM code, beyond the relevant classes
> implementing "Runnable" and/or "Callable").
>>> Some ideas were floated (cf. ML archive) but no implementation or
>>> experiment...  Perhaps with a well-defined goal such as performance
>>> improvement, your design suggestions will become clearer to more people.
>>> AFAIK, only the classes in the "o.a.c.m.neuralnet" package are currently
>>> ready to be used with the "java.util.concurrent" framework.
>> FWIU Neural Nets are a great fit for concurrency.
> Quite true.
> But even the optimizers could benefit from just being able to use
> more threads: It is often (always?) necessary to evaluate the objective
> function "N" times per iteration. So, the computation could be about
>   min(N, numCores)
> times faster.
>> I think for the
>> others we will end up having discussions around how users would
>> control the number of threads, etc. again that makes some of us
>> nervous.
> One additional parameters: numCores.
>> An asynchronous operation that runs in one separate thread
>> is easier to reason about.
> Sure.
> But then we should stop talking about performance on this list. ;-}
>> If we want to test 10 neural net
>> configurations, and we have 10 cores, then we can start each by itself
>> by doing something like:
>> Nework.setAsync(true).addNeurons().connectNeurons().addObserver(observer).start().
>> //Now do 10 more
>> //If the observer is shared then notifications should be thread safe.
> I had a similar argument for not making "FastMath" initialization faster
> (at the cost of a lot of additional code):  It was rejected...
> Regards,
> Gilles
> P.S. I think that several issues evoked in this thread could warrant opening
>      their own thread, to gather more opinions on actual actions to be taken.
>> Cheers,
>> - Ole
>> P.S. Dang that was a long email.  If I write one more of these, ban me :)
> My fault: I should not keep answering! ;-)
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message