lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Sonawane <pankaj4sonaw...@gmail.com>
Subject Solr limiting number of rows to indexed to 21500 every time.
Date Tue, 13 Jan 2015 07:31:24 GMT
Hi,

I am using Solr DataImportHandler to index data from database
table(Oracle). One of the column contains String representation of XML
(Sample below).

*<options>*
*<option name="A">1</option>*

*<option name="B">2</option>*

*<option name="C">3</option>*
*.*
*.*
*.*

*</options> //<option> can be 100-200*

I want solr to index each 'name' in 'option' tag against its value

ex. JSON for 1 row
"docs": [ {
"COL1": "F",
"COL2": "ASDF", "COL3": "ATCC", "COL4": 29039757, "A_s": "1", "B_s": "2", "
C_s": "3",
.
.
.
*          }*
// appending '_s' to 'name' attribute for making dynamic fields.


But while indexing data, *every time only 21500 rows get indexed*. After
these much records get indexed I got following exception:

*1320927 [Thread-15] ERROR
org.apache.solr.handler.dataimport.EntityProcessorBase  û getNext() failed
for query 'SELECT col1,col2,col3,col4,XMLSERIALIZE(col5 AS  CLOB) AS col5
FROM
tableName':org.apache.solr.handler.dataimport.DataImportHandlerException:
java.sql.SQLRecoverableException: No more data to read from socket*
*        at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:63)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:378)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:258)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:293)*
*        at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:116)*
*        at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)*
*        at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)*
*        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)*
*        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)*
*        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)*
*Caused by: java.sql.SQLRecoverableException: No more data to read from
socket*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1200)*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(T4CMAREngine.java:1865)*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(T4CMAREngine.java:1757)*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(T4CMAREngine.java:1750)*
*        at
oracle.jdbc.driver.T4CClobAccessor.handlePrefetch(T4CClobAccessor.java:543)*
*        at
oracle.jdbc.driver.T4CClobAccessor.unmarshalOneRow(T4CClobAccessor.java:197)*
*        at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:916)*
*        at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:835)*
*        at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:664)*
*        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:328)*
*        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)*
*        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)*
*        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:194)*
*        at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1074)*
*        at
oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:369)*
*        at
oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:273)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:370)*
*        ... 12 more*

*1320928 [Thread-15] ERROR org.apache.solr.handler.dataimport.DocBuilder  û
Exception while processing: e1 document : SolrInputDocument(fields:
[]):org.apache.solr.handler.dataimport.DataImportHandlerException:
java.sql.SQLRecoverableException: No more data to read from socket*
*        at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:63)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:378)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:258)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:293)*
*        at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:116)*
*        at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)*
*        at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)*
*        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)*
*        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)*
*        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)*
*        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)*
*Caused by: java.sql.SQLRecoverableException: No more data to read from
socket*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1200)*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(T4CMAREngine.java:1865)*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(T4CMAREngine.java:1757)*
*        at
oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(T4CMAREngine.java:1750)*
*        at
oracle.jdbc.driver.T4CClobAccessor.handlePrefetch(T4CClobAccessor.java:543)*
*        at
oracle.jdbc.driver.T4CClobAccessor.unmarshalOneRow(T4CClobAccessor.java:197)*
*        at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:916)*
*        at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:835)*
*        at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:664)*
*        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:328)*
*        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)*
*        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)*
*        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:194)*
*        at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1074)*
*        at
oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:369)*
*        at
oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:273)*
*        at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:370)*
*        ... 12 more*


*db-data-config.xml:*

<dataConfig>
 <script><![CDATA[
                function makePair(row) {
  var theKey = row.get("theKey")+ "_s";
  var theValue = row.get("theValue");
  row.put(theKey, theValue);
  row.remove("theKey");
  row.remove("theValue");
  return row;
  }
        ]]></script>

    <dataSource name="a1" driver="oracle.jdbc.driver.OracleDriver"
url="***" user="***" password="***"/>
    <dataSource name="a2" type="FieldReaderDataSource" encoding="UTF-8"/>

    <document>
        <entity name="e1" query="SELECT col1,col2, col3,col4,
        XMLSERIALIZE(col5 AS  CLOB) AS col5 FROM tableName"
                 dataSource="a1" pk="col1">
            <field column="col1" name="col1" />
            <field column="col2" name="col2" />
            <field column="col3" name="col3" />
            <field column="col4" name="col4" />
            <entity name="e2" dataSource="a2"
processor="XPathEntityProcessor"
            dataField="e1.col5"
            forEach="/options/option"
             transformer="script:makePair">
      <field column="theKey" xpath="/options/option/@name" />
      <field column="theValue" xpath="/options/option/value" />
      </entity>
        </entity>
    </document>
</dataConfig>

Please Help me to resolve this issue.

Thanks,
Pankaj

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message