commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject Re: [MATH] Matrix hierarchy reorganization
Date Mon, 21 Feb 2011 13:16:51 GMT
Hi Mikkel,

----- "Mikkel Meyer Andersen" <> a écrit :

> Dear community,
> I've started on implementing Kalman filter and have once again run
> into a structural challenge in regards to matrix hierarchy (as with
> and
> ).
> We have two base types of matrices: RealMatrix and FieldMatrix, both
> extending AnyMatrix. Both RealMatrix and FieldMatrix have a lot of
> the
> same functionality, e.g. add, subtract, scalarAdd, scalarMultiply,
> multiply, preMultiply etc.
> I think it would be beneficial to try to move these standard matrix
> methods to AnyMatrix, or at least introduce a new intermediate
> AnyAlgebraicMatrix (or similar). It would be natural in several
> aspects to just require an AnyAlgebraicMatrix, e.g. with Kalman
> filter
> or repeated squaring, instead of forcing the user to use either
> RealMatrix or FieldMatrix (or implement almost similar code for
> both).
> What do you think about such a reorganization of the matrix
> hierarchy?

I think we really need to reorganize our linear algebra, but am not sure about
this specific change, as I fear it would deter performances (which already
needs improvements).

At ApacheCon Atlanta, we had a short discussion with Isabel Drost form the Mahout
project. We would really much have liked to have Ted Tunning around too
but he was not there. We said to Isabel that once we have 2.2 out and
will be reflecting about 3.0, we would like to have some requirements from
the Mahout project. We were not able up to now to satisfy their needs because
of backward compatibility. However, now is the time we can change anything
we want.

The current problems I see with our linear algebra package are the following ones:
 - clumsy hierarchy
 - inefficient for large matrices (say above 4000 rows and columns)
 - insufficient support for sparse matrices (most algorithms simply don't use sparsity)
 - SVD is still not stable, after 3 complete rewrites
 - missing features (eigen decomposition for non-symetric matrices)

The cool features we have are:
 - support for field matrices (this is really useful with Complex elements,
   with Fraction elements when we need exact computation of polynomial coefficients for
   example, and with Dfp elements for high accuracy)
 - reasonable API for decomposition, mainly Jama compatible but slightly better IMHO

The things we can change at 3.0 are:
 - API
 - implementation

The things we will for sure have some difficult discussions are:
 - dependencies
   (I guess Phil and myself don't like dependencies, whereas Ted and Gilles
    are happy with them)

What I would like to see:
 - more exchanges with the Mahout community

I'm not sure if we should have this discussion here, or on the Mahout list or
on both. Ted, what to you think about this, are their a sufficient number of
Mahout people here or should we cross-post everything ?

Thanks for bringing up this really important topic Mikkel!

> Cheers, Mikkel.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message