lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avlesh Singh <>
Subject Re: Data loading from DB - data sizes and obstacles
Date Fri, 07 Aug 2009 06:07:53 GMT
I have been a satisfied DIH user for a long time.
The project I use Solr for, runs on a MySQL (5.1) version. There are 6
solr-cores in total with a combined index size of 12G. The database design
is as relational as it can get, and writing SQL queries to fetch the data
has always been always a problem. Thanks to DIH, I have honed my DB concepts
and created nice procedure and views to faltten out data.
For DIH, I have kept it simple select statements (or procedure calls) with a
few entities and pushed the heavylifting to database(scripts).

I am talking about 4million records here.
I have never tried LuSQL.


On Fri, Aug 7, 2009 at 11:15 AM, Amit Nithian <> wrote:

> All,
> An off and on project of mine has been to work on refactoring the way we
> load data from MySQL into Solr. Our current approach is fairly hard coded
> and not configurable as I would like. I was curious of people who have used
> the DIH and/or LuSQL to load data into Solr, how much data you typically
> load and obstacles you have run into along the way. For example, some of
> our
> SQL queries are quite complex with a bunch of joins which can cause
> headaches for the DB.
> I am mainly curious for those who use MySQL for comparison. I am also
> looking to evaluate DIH vs LuSQL (the 0.9.3 which I read about but haven't
> seen for download). Has any progress been made on the making DIH a separate
> library?
> Sorry for the flurry of questions but am interested in everyones response!
> Thanks
> Amit

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message