mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Possible contribution at somewhat of a tangent to Mahout
Date Sun, 04 Oct 2009 18:49:57 GMT
The algorithm in question is a very interesting and *very* useful one for
language processing and understanding.  As Benson says, it is non-trivial to
implement.

I would be happy to kibitz on a re-implementation as usual, but have
negative time available for real coding.  Is anybody else available?

On Sun, Oct 4, 2009 at 5:25 AM, Benson Margulies <bimargulies@gmail.com>wrote:

> Isabel,
>
> Let me give you the backrground of this. The code in question is a
> second-generation implementation (by the same author) of this very hard
> algorithm.The author is  a professional who now works for Basis (where I
> work) who was working, at the time, at an academic institution. I have
> opened negotiations with the relevant professor to see if I can't find a
> way
> to make it open source.
>
> Last year, before I knew anything about this implementation, I built one.
> Mine, written in C++, can turn 111921 words with 8899115 distinct bigrams
> into 1000 clusters in about 14 hours on a 4-core system. While I'm a fairly
> experienced C++ code tuner, I am not much of a mathematician, and so I
> missed some mathematical shortcuts.
>
> I used OpenMP to get some parallelization into the code. However, it seems
> to me that some major rethinking would be required to cast the problem in a
> form where Java and Hadoop could do anything with it. However, I could just
> be thick-headed.
>
> If I succeed in getting the code to open source, I'd be very interested in
> seeing if any of you hadoop-artists can suggest an approach that could
> produce comparable or better clock time without having to apply a gigantic
> amount of hardware. GIven such an approach, the author or I might find
> interesting to try an implementation an a contribution to Mahout.
>
> --benson
>
>
> On Sun, Oct 4, 2009 at 7:54 AM, Isabel Drost <isabel@apache.org> wrote:
>
> > On Saturday 03 October 2009 18:45:12 Sean Owen wrote:
> > > Let me however revive my suggestion that Mahout include a 'sandbox'
> > > module of sorts to host anything at all. This neatly allows for
> > > incorporation of anything, in any state, without confusing users as to
> > > what should be expected of Mahout 'proper', which should be a
> > > reasonably high bar come version 1.0.
> >
> > +1 Until that is realized, I would suggest to not scare away people just
> > because they used the "wrong" programming language/lib/...
> >
> > Benson, do you think there might be a tiny chance that you can motivate
> the
> > student to contribute his implementation as a JIRA issue and work
> together
> > with the community to make it run on Hadoop? Does that even make sense
> for
> > the algorithm implemented?
> >
> > Isabel
> >
> >
> > --
> >  |\      _,,,---,,_       Web:   <http://www.isabel-drost.de>
> >  /,`.-'`'    -.  ;-;;,_
> >  |,4-  ) )-,_..;\ (  `'-'
> > '---''(_/--'  `-'\_) (fL)  IM:  <xmpp://MaineC.@spaceboyz.net>
> >
> >
>



-- 
Ted Dunning, CTO
DeepDyve

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message