lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From madhav bahuguna <madhav.bahug...@gmail.com>
Subject solr indexing not working when i try to insert 1000000 rows but works fine when i try to index 400000 rows or below
Date Tue, 03 Jun 2014 06:00:18 GMT
HI

iam using solr 4.7.1 and trying to do a full import.My data source is a
table in mysql. It has 10000000 rows and 20 columns.

Whenever iam trying to do a full import solr stops responding. But when i
try to do a import with a limit of 400000 or less it works fine.

If i try to import more than this solr wont index the result it either
stops responding or will show "indexing failed". In the error log it says
"Unable to execute query".But i dont understand how is the query running
fine for lesser number of records but fails when i run more number of
records

My system config are follows

CPU-i7 Ram -6Gb OS-64 bit windows 7 I am not able to figure out what the
problem is ,i have tried increasing the max_allowed_packet to 1000M and
even java heap size.

please help thanks in advance

*This is the error code*

`Exception while processing: playername document :
SolrInputDocument(fields:
[]):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable
to execute query: SELECT
player_id,firstname,lastname,value1,value2,value3,value4,value5,value6,
value7,value8,value9,value10, value11,value18,value19,value20, country_id,
playername_modtime,player_flag from playername WHERE 'true' != 'false' OR
playername.playername_modtime > '2014-05-23 10:38:56' Processing Document #
1 at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:281)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:238)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:42)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:477)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:331)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:239)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure

The last packet successfully received from the server was 130,037
milliseconds ago. The last packet sent successfully to the server was
130,038 milliseconds ago. at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source) at
com.mysql.jdbc.Util.handleNewInstance(Util.java:409) at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1127)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2288) at
com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2044) at
com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3549) at
com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:489) at
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3240) at
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2411) at
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2834) at
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2832) at
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2781) at
com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:908) at
com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:788) at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:274)
... 12 more Caused by: java.io.EOFException: Can not read response from
server. Expected to read 6 bytes, read 4 bytes before connection was
unexpectedly lost. at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3161)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2269) ... 23 more
5/23/2014 8:32:18 PM ERROR DataImporter Full Import
failed:java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
execute query: SELECT
player_id,​firstname,​lastname,​value1,​value2,​value3,​value4,​value5,​value6,​
value7,​value8,​value9,​value10,​ value11,​value18,​value19,​value20,​
country_id,​ playername_modtime,​player_flag from playername WHERE 'true'
!= 'false' OR playername.playername_modtime > '2014-05-23 10:38:56'
Processing Document # 1 Last Check: 5/23/2014 8:36:34 PM`


One more thing i have noticed is that when i try to do a full import with
just  2 columns and increase the jvm size(java -Xmx1024m -jar start.jar) it
works fine.

Iam new to solr and Iam not able to figure out what the issues is.

Please advise and help

Regards

Madhav Bahuguna

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message