lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "郭芸" <mickey.guo...@gmail.com>
Subject Re: Re: About Solr DataImportHandler
Date Wed, 08 Sep 2010 02:45:56 GMT
Thank you for your reply,it is very import to me.
1.I  agree with you by i read solr's source code,i found that it can resolve this problem
by config db-data-config.xml,like this(my database's Sqlserver2005,other database will unavailable):
<dataSource name="dsSqlServer" type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
batchSize="3000"
        url="jdbc:sqlserver://192.168.1.5:1433; DatabaseName=testDatabase;responseBuffering=adaptive;selectMethod=cursor"
user="sa" password="12345" />

add  responseBuffering=adaptive;selectMethod=cursor into url attribute,and Solr will set these
parameters by itself:
c.createStatement(ResultSet.TYPE_FORWARD_ONLY,ResultSet.CONCUR_READ_ONLY);
By these configs,Solr can import big table's datas to  index dir.


2.But there are some problems:
if the table is very big,solr will spend a long time to import and index,may be one day and
more.so once occurred network problems and others during this time,maybe solr can not remember
what documents had processed,and if we continue data import ,we do not know where to start.

3.i am sorry for my bad English.i wish you can know what i mean.


2010-09-08 



郭芸 



发件人: Alexey Serba 
发送时间: 2010-09-07  16:07:49 
收件人: dev 
抄送: 
主题: Re: About Solr DataImportHandler 
 
> i found that Solr import the datas to memory first,then write them to index dir.
That's not really true. DataImportHandler streams the result from
database query and adding documents into index. So it shouldn't load
all database data into memory. Disabling autoCommit, warming queries
and spellcheckers usually decreases required amount of memory during
indexing process.
Please share your hardware details, jvm options, solrconfig and schema
configuration, etc.
2010/9/7 郭芸 <mickey.guoyun@gmail.com>:
> Dear all:
> I use Solr DataImportHandler's JdbcDataSource to import the Sqlsever 2005's
> datas to Solr,but My table is versy big,about 300G.and i found that Solr
> import the datas to memory first,then write them to index dir.So if the
> datas are too big,there will trigger an OutOfMemoryException.
> I want to solve this problem,and how can ti do it?anybody can help me?Thank
> you.
>
> 2010-09-07
> ________________________________
> 郭芸
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Mime
View raw message