mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florent Empis <florent.em...@gmail.com>
Subject Re: Could you improve the AbstractJDBCDataModel?
Date Fri, 30 Jul 2010 14:30:04 GMT
Actually, I discussed a while back the creation of a caching strategy for
the datamodel, through Memcache.
I did begin an implementation, but other stuff (namely, a new job) got in
the way....

Another option would be to go for a library such as Hibernate or Ibatis and
throw efficient caching (such as EhCache) at it.
You'd maybe loose some speed (particularly with the added abstraction layer
of Hibernate), but it would probably not be too big of job to implement
(particularly with Ibatis, where it would be a matter a few days: take the
existing statements, convert them to plain sql request, write caching
configuration and you're all set)


2010/7/29 Sean Owen <srowen@gmail.com>

> That's rather the point of the JDBC-backed models -- they're for the
> case where you can't load data in memory. If you can, then load them
> in memory.
>
> Yes it's not fast to use JDBC. It only really makes sense with
> algorithms that, through their nature or caching, don't read a lot of
> data. You want to use the Caching* wrappers everywhere.
>
> You might write some kind of DataModel wrapper which caches some
> number of user preferences to mitigate this effect, yes.
>
> 2010/7/29 Young <woshidustin@126.com>:
> > Hi All,
> > I use MySQLJDBCDataModel for ItemBasedRecommender for online
> recommendation. I have put other stuff in the memory and do the
> itemsimilarity matrix precomputation.
> >
> > I find when I do one single recommendation, the
> getPreferencesFromUser(long id) method (in )will be called thousands of
> times. I think it is okay when using in-memory datamodel. But if it is in
> the AbstractJDBCDataModel, it will take much more time because the *same*
> mysql queries will be excuted thousands of times.
> >
> > Thanks.
> > ---Young
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message