lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Build suggester in different directory (not /tmp).
Date Wed, 20 Dec 2017 20:47:53 GMT
Matthew:

I think you'll be awfully unhappy with buildOnCommit. Say you're
bulk-indexing and committing every 15 seconds....

buildOnStartup is problematical as well since it'd rebuild everytime
you bounced Solr even if the index hadn't changed.

Personally I'd alter my indexing process to fire a build command when
it was done.

Or, if you can afford to optimize after _every_ set of updates (say
you only update every day or less often) then buildOnOptimize makes
sense.

Best,
Erick

On Wed, Dec 20, 2017 at 12:40 PM, Matthew Roth <mgroth86@gmail.com> wrote:
> Thanks Erick,
>
> I'll head your warning. Ultimately, the index will be rather static so I do
> not fear much from buildingOnComit. But I think building on startup would
> likely be set to false regardless.
>
> Shawn,
>
> Thank you as well. That is very informative regarding java.io.tmpdir. I am
> starting this as a service, but I think I can handle making the required
> changes.
>
> Best,
> Matt
>
> On Wed, Dec 20, 2017 at 2:58 PM, Shawn Heisey <apache@elyograg.org> wrote:
>
>> On 12/20/2017 10:05 AM, Matthew Roth wrote:
>> > I am building a few suggester's and I am receiving the error that I have
>> no
>> > space left on device.
>>
>> <snip>
>>
>> > At first this threw me. df showed I had over 100 G free. the /data dir
>> the
>> > suggester is being constructed from is only 4G. On a subsequent run I
>> > notice that the suggester is first being built in /tmp. When setting up
>> the
>> > LVM I only allotted 2g's to that directory and I prefer to keep it that
>> > way.
>>
>> The code is utilizing the "java.io.tmpdir" system property to determine
>> a temporary directory location to use for the build, before it is put in
>> the final location.  On POSIX platforms, this will default to /tmp.
>>
>> If you are starting Solr manually, then you would just need to add the
>> following parameter to the bin/solr commandline (including the quotes)
>> to change this location:
>>
>> -a "-Djava.io.tmpdir=/other/tmp/path"
>>
>> If you've installed Solr as a service, then I do not think there's any
>> easy way to adjust this property, other than manually editing bin/solr
>> to add the -D option to the startup commandline.  We'll need an
>> enhancement issue in Jira to modify the script so it can set
>> java.io.tmpdir from an environment variable.
>>
>> Note that adjusting this property may result in other things that Solr
>> creates being moved away from /tmp.
>>
>> Since most POSIX operating systems will automatically delete old files
>> in /tmp, it's always possible that when you move Java's temp directory,
>> you'll end up with cruft in the new location that never gets deleted.
>> Developers do generally try to clean up temporary files, but sometimes
>> things go wrong that weren't anticipated.  If that does happen and a
>> temporary file is created by Lucene/Solr that doesn't get deleted, then
>> I would consider that a bug that should be fixed.
>>
>> On Windows systems, Java asks the OS where the temp directory is.  The
>> info I've found says that the TMP environment variable will override
>> this location for Windows, but not for other platforms.
>>
>> Thanks,
>> Shawn
>>
>>

Mime
View raw message