Luc,
Agreed. Dimitri brought out a good point about correlated errors that I didn't think about.
All these cases can be handled by user modifying his/her input, but it would be nice to think
about how this can be supported in the future. I think this problem showed up for two general
reasons, which I brought out before.
1) We are not testing our optimization classes on large datasets.
2) The linear algebra framework should be used internally, that way we can leverage sparse
matrix operations.
Konstantin
On Dec 28, 2012, at 10:48 AM, Luc Maisonobe <luc@spaceroots.org> wrote:
> Le 28/12/2012 16:08, Dimitri Pourbaix a écrit :
>> Luc,
>>
>>> However, it is also possible to set a nondiagonal weight matrix, and
>>> one class (AbstractLeastSquaresOptimizer) performs an eigen dcomposition
>>> on the matrix to extract its square root. I don't know any use case for
>>> this, but it has been set up this way, so I guess someone has a use for
>>> nondiagonal weights.
>>
>> Such a situation occurs when observations are correlated. That is
>> actually the most general expression for a least square problem.
>>
>>> I wonder if I should simply add this as is or if we should rather remove
>>> the nondiagonal weights feature and support only vector weights.
>>
>> Even if a vector of weights is convenient, it would only cover a subset
>> of situations. However, even a vector of weights is not needed if both
>> the models and the observations are premultiplied by the square root
>> of their weight. By the way, I remind you that those weights already
>> caused some bugs in the 2.0 release.
>>
>> Personnally, I could live with a vector form.
>
> So in order to make sure I understand your point, you would be OK if I
> deprecate the nondiagonal weights, in which case users needing this
> would have to implement it themselves by premultiplication (as both you
> and Konstantin seem to propose)?
>
>>
>> As a more general comment, I find it amazing that all the +1 for the
>> release were only concerned by the compliance with (commons) rules,
>> configuration files, ... Just 4 days after the release, you suddenly
>> figure out that a user is in trouble and you want a quick fix. Maybe
>> such a test would have been need BEFORE the release!
>
> Sure, but for the record the feature was also a last minute change. This
> was discussed on the list, and the final decision was to add this
> feature despite the release was close. No wonder we failed to test it
> thoroughsly.
>
> We don't expect our releases to be perfect. We do our best, with the
> resources we have.
>
>>
>> Regards,
>> Dim.
>
>
> Le 28/12/2012 16:13, Konstantin Berlin a écrit :
>> Hi,
>>
>> I am not clear on the use of weights in a nonleast squares problem
>> is. In a leastsquares problem the weights can simply be taken in by
>> the user in their input function. So the weights option is a
>> convenience feature, any nonstandard matrix weights could be
>> programmed in by the user, if they need to. I have a hard time
>> imagining a case when the weights are nondiagnal. I think your idea
>> is good and you should go on with it. The quadratic memory scaling
>> with the number of data points is not acceptable.
>
> Thanks Konstantin. So if Dimitri confirm I understood his point
> correctly, I will proceed with deprecating the Weight class and
> introduce a WeightVector class, understanding it is a convenience
> feature suited for simple noncorrelated observations.
>
>>
>> Konstantin
>
> best regards,
> Luc
>
> 
> To unsubscribe, email: devunsubscribe@commons.apache.org
> For additional commands, email: devhelp@commons.apache.org
>

To unsubscribe, email: devunsubscribe@commons.apache.org
For additional commands, email: devhelp@commons.apache.org
