lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Hirschfeld <tomhirschf...@gmail.com>
Subject EOF exception from ramDirectory search in spark
Date Fri, 11 May 2018 02:19:00 GMT
Hey All,
I have a fun issue I'm dealing with at the junction of lucene and spark.

I have an RDD[(key, iterator1, iterator2)]

I run a mapPartitions on the RDD, and for each partition, I create a
ramDirectory, I index all of the elements in interator1, and then I search
the index for each element in iterator2. The issue that I am having is all
of my searches on the ramDirectory fail with an "EOF exception" Here is an
example of one of the EOF exceptions:

java.lang.RuntimeException: java.io.EOFException: seek beyond EOF:
pos=69377 vs length=53924:
RAMInputStream(name=RAMInputStream(name=_1bjl_Lucene54_0.dvd)
[slice=randomaccess]), java.lang.RuntimeException: java.io.EOFException:
seek beyond EOF: pos=98833 vs length=48835:



To recap: each executor loops through, create a ram directory, writes to
it, and then reads from it.


I have been trying for the past few days to address this issue but I have
been unable to find out whats going on. Any hint as to what might be
happening here?

Best,
Tom Hirschfeld

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message