lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Lu <chris...@gmail.com>
Subject Re: indexing database
Date Wed, 21 Jan 2009 21:36:20 GMT
This is not a lucene question, but a jdbc question.
The code is not releasing the jdbc connection, statement, and resultset, and
what's worse, the code is creating new connections when paginating the
results.

-- 
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
DBSight customer, a shopping comparison site, (anonymous per request) got
2.6 Million Euro funding!

On Wed, Jan 21, 2009 at 1:29 PM, cemsoft <bcemkucuk@yahoo.com> wrote:

>
> hi,
>
> i am trying to index a database table with appr. 1,5 million rows..
> first i got OutOfMemory exception, then i used offsets, it works now till
> 50000 rows
>
> lucene is new to me ..should i set sth else ?? below is the code that i use
>
> private static void indexData(MysqlDataSource ds) throws SQLException,
> Exception {
>               JdbcDirectory jdbcDir = new JdbcDirectory(ds, new
> MySQLDialect(), "indexTable");
>                jdbcDir.create(); // creates the indexTable in the DB
> (test). No need to
>                                                        // create it
> manually
>                StandardAnalyzer analyzer = new StandardAnalyzer();
>
>                int offset=0;
>                int rowCount = 5000;
>
>                IndexWriter writer = new IndexWriter(jdbcDir, analyzer,
> true,
> MaxFieldLength.UNLIMITED);
>                try{
>                Connection conn = ds.getConnection();
>                while (offset<1244800){
>                        indexDocs(writer, conn, offset, rowCount);
>                        offset += rowCount;
>                }
>                System.out.println("Optimizing...");
>                writer.optimize();
>                writer.close();
>                }catch (Exception e) {
>                        System.out.println("exception " + e.getMessage());
>                }
>
>        }
>
>        static void indexDocs(IndexWriter writer, Connection conn, int
> offset, int
> rowCount) throws Exception {
>
>                String sql = "select id, content from
> monitoring_dk_test.entry limit "+
> offset + ","+ rowCount+";";
>                System.out.println(sql);
>                ResultSet rs = conn.createStatement().executeQuery(sql);
>                while (rs.next()) {
>                        Document d = new Document();
>                        d.add(new Field("id", rs.getString("id"),
> Field.Store.YES,
> Field.Index.NO));
>                        d.add(new Field("content", rs.getString("content"),
> Field.Store.YES,
> Field.Index.TOKENIZED));
>                        writer.addDocument(d);
>                }
>                System.out.println("indexing finished ...");
>        }
>
>
>
> best regards
> cem
> --
> View this message in context:
> http://www.nabble.com/indexing-database-tp21592598p21592598.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message