lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Sturlese <marc.sturl...@gmail.com>
Subject Re: Deduplication patch not working in nightly build
Date Mon, 05 Jan 2009 13:05:52 GMT

Thanks I will have a look to my JdbcDataSource. Anyway it's weird because
using the 1.3 release I don't have that problem...

Shalin Shekhar Mangar wrote:
> 
> Yes, initially I figured that we are accidentally re-using a closed data
> source. But Noble has pinned it right. I guess you can try looking into
> your
> JDBC driver's documentation for a setting which increases the connection
> alive-ness.
> 
> On Mon, Jan 5, 2009 at 5:29 PM, Noble Paul നോബിള്‍ नोब्ळ् <
> noble.paul@gmail.com> wrote:
> 
>> I guess the indexing of a doc is taking too long (may be because of
>> the de-dup patch) and the resultset gets closed automaticallly (timed
>> out)
>> --Noble
>>
>> On Mon, Jan 5, 2009 at 5:14 PM, Marc Sturlese <marc.sturlese@gmail.com>
>> wrote:
>> >
>> > Donig this fix I get the same error :(
>> >
>> > I am going to try to set up the last nigthly build... let's see if I
>> have
>> > better luck.
>> >
>> > The thing is it stop indexing at the doc num 150.000 aprox... and give
>> me
>> > that mysql exception error... Without DeDuplication patch I can index 2
>> > milion docs without problems...
>> >
>> > I am pretty lost with this... :(
>> >
>> >
>> > Shalin Shekhar Mangar wrote:
>> >>
>> >> Yes I meant the 05/01/2008 build. The fix is a one line change
>> >>
>> >> Add the following as the last line of DataConfig.Entity.clearCache()
>> >> dataSrc = null;
>> >>
>> >>
>> >>
>> >> On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
>> >> <marc.sturlese@gmail.com>wrote:
>> >>
>> >>>
>> >>> Shalin you mean I should test the 05/01/2008 nighlty? maybe with this
>> one
>> >>> works? If the fix you did is not really big can u tell me where in
>> the
>> >>> source is and what is it for? (I have been debuging and tracing a lot
>> the
>> >>> dataimporthandler source and I I would like to know what the
>> imporovement
>> >>> is
>> >>> about if it is not a problem...)
>> >>>
>> >>> Thanks!
>> >>>
>> >>>
>> >>> Shalin Shekhar Mangar wrote:
>> >>> >
>> >>> > Marc, I've just committed a fix which may have caused the bug.
Can
>> you
>> >>> use
>> >>> > svn trunk (or the next nightly build) and confirm?
>> >>> >
>> >>> > On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള്‍
नोब्ळ् <
>> >>> > noble.paul@gmail.com> wrote:
>> >>> >
>> >>> >> looks like a bug w/ DIH with the recent fixes.
>> >>> >> --Noble
>> >>> >>
>> >>> >> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese
>> >>> <marc.sturlese@gmail.com>
>> >>> >> wrote:
>> >>> >> >
>> >>> >> > Hey there,
>> >>> >> > I was using the Deduplication patch with Solr 1.3 release
and
>> >>> >> everything
>> >>> >> was
>> >>> >> > working perfectly. Now I upgraded to a nigthly build (20th
>> december)
>> >>> to
>> >>> >> be
>> >>> >> > able to use new facet algorithm and other stuff and
>> DeDuplication
>> is
>> >>> >> not
>> >>> >> > working any more. I have followed exactly the same steps
to
>> apply
>> >>> the
>> >>> >> patch
>> >>> >> > to the source code. I am geting this error:
>> >>> >> >
>> >>> >> > WARNING: Error reading data
>> >>> >> > com.mysql.jdbc.CommunicationsException: Communications
link
>> failure
>> >>> due
>> >>> >> to
>> >>> >> > underlying exception:
>> >>> >> >
>> >>> >> > ** BEGIN NESTED EXCEPTION **
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> >
>> >>> >> > STACKTRACE:
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> >        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> >>> >> >        at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> >        at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> >        at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> >        at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> >
>> >>> >> >
>> >>> >> > ** END NESTED EXCEPTION **
>> >>> >> > Last packet sent to the server was 202481 ms ago.
>> >>> >> >        at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> >        at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> >        at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> >        at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> > Jan 5, 2009 10:06:16 AM
>> >>> >> org.apache.solr.handler.dataimport.JdbcDataSource
>> >>> >> > logError
>> >>> >> > WARNING: Exception while closing result set
>> >>> >> > com.mysql.jdbc.CommunicationsException: Communications
link
>> failure
>> >>> due
>> >>> >> to
>> >>> >> > underlying exception:
>> >>> >> >
>> >>> >> > ** BEGIN NESTED EXCEPTION **
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> >
>> >>> >> > STACKTRACE:
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> >        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> >>> >> >        at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> >        at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> >        at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> >        at
>> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> >>> >> >        at
>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> >>> >> >        at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> >
>> >>> >> >
>> >>> >> > ** END NESTED EXCEPTION **
>> >>> >> >
>> >>> >> > Last packet sent to the server was 202482 ms ago.
>> >>> >> >        at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> >        at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> >        at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> >        at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> >        at
>> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> >>> >> >        at
>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> >>> >> >        at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> >        at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> >
>> >>> >> > Mark do you know if I should to some extra hack to run
>> DeDuplication
>> >>> >> patch
>> >>> >> > in the new solr version (nightly)? Has something changed
in the
>> >>> source
>> >>> >> that
>> >>> >> > make it not work or it's simply that i am doing something
wrong?
>> >>> >> >
>> >>> >> > Thanks in advanced
>> >>> >> > --
>> >>> >> > View this message in context:
>> >>> >>
>> >>>
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>> >>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >>> >> >
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> --Noble Paul
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Regards,
>> >>> > Shalin Shekhar Mangar.
>> >>> >
>> >>> >
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Shalin Shekhar Mangar.
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>>
>>
>>
>> --
>> --Noble Paul
>>
> 
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 
> 

-- 
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21290661.html
Sent from the Solr - User mailing list archive at Nabble.com.


Mime
View raw message