lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (Commented) (JIRA)" <>
Subject [jira] [Commented] (SOLR-2864) DataImportHandler has non-deterministic sort order for XML files
Date Thu, 17 Nov 2011 20:07:51 GMT


Hoss Man commented on SOLR-2864:

Gabriel: one thing that's not clear to me from your patch is the interplay between the deterministic
sorting and the recursion: if i'm reading this right, directories are sorted deterministically
by modification date, then they are walked recursively -- as opposed to doing a recursive
walk, then sorting all the matching files by date. i have no opinion wether that's good or
bad, but it seems worthy of consideration, testing, and documentation.

minor nits:

1) if the goal is to be deterministic, then *only* sorting on last mod date doesn't seem like
enough -- there should also be a secondary sort on something guaranteed to be unique (like
fullpath or name) correct?

2) the CREATED_FIRST and CREATED_SECOND filename constants in your tests should be used consistently,
there's no reason for multiple "a.xml" and "b.xml" strings in the test.  using the constants
everywhere will help make it clear that those files are being created in a specific order
for a specific reason.
> DataImportHandler has non-deterministic sort order for XML files
> ----------------------------------------------------------------
>                 Key: SOLR-2864
>                 URL:
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 3.4
>            Reporter: Gabriel Cooper
>            Priority: Minor
>              Labels: dataimport, patch, xml
>             Fix For: 3.5
>         Attachments: lucene-2864.patch
>   Original Estimate: 1h
>  Remaining Estimate: 1h
> DataImportHandler's FileListEntityProcessor relies on Java's File.list() method to retrieve
a list of files from the configured dataimport directory, but list() does not guarantee a
sort order ^(1)^. This means that if you have two files that update the same record, the results
are non-deterministic. Typically, list() does in fact return them lexigraphically sorted,
but this is not guaranteed ^(2)^.
> An example of how you can get into trouble is to imagine the following:
> xyz.xml -- Created one hour ago. Contains updates to records "Foo" and "Bar".
> abc.xml -- Created one minute ago. Contains updates to records "Bar" and "Baz".
> In this case, the newest file, in abc.xml, would (likely, but not guaranteed) be run
first, updating the "Bar" and "Baz" records. Next, the older file, xyz.xml, would update "Foo"
and overwrite "Bar" with outdated changes.
>  (1) Per,5,0/docs/api/java/io/File.html#list%28%29
> "There is no guarantee that the name strings in the resulting array will appear in any
specific order; they are not, in particular, guaranteed to appear in alphabetical order."
>  (2)  Even if it was guaranteed, lexigraphical sorting would give you the following sort
>   1.xml
>   10.xml
>   2.xml
>   ...

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message