lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: solr 7.0: What causes the segment to flush
Date Tue, 17 Oct 2017 23:40:36 GMT
bq:  Is there a way to not write to disk continuously and only write the file...

Not if we're talking about the transaction log. The design is for the
transaction log in particular to continuously get updates flushed to
it, otherwise you could not replay the transaction log upon restart
and have any hope of not losing data.

And one other thing you want to be aware of: Having such long delays
between commits will mean that in the event of ungraceful shut-down
(pull the plug, kill -9 and the like) the _entire_ set if documents
sent since the last time you did a commit will be replayed from the
tlog before the replica accepts incoming requests.

As for <2> I haven't a clue.

Best,
Erick

On Tue, Oct 17, 2017 at 8:40 AM, Nawab Zada Asad Iqbal <khichi@gmail.com> wrote:
> I take my yesterday's comment back. I assumed that the file being written
> is a segment, however after letting solr run for the night. I see that the
> segment is flushed at the expected size:1945MB (so that file which i
> observed was still open for writing).
> Now, I have two other questions:-
>
> 1. Is there a way to not write to disk continuously and only write the file
> when segment is flushed?
>
> 2. With 6.5: i had ramBufferSizeMB=20G and limiting the threadCount to 12
> (since LUCENE-6659
> <https://issues.apache.org/jira/plugins/servlet/mobile#issue/LUCENE-6659>,
> there is no configuration for indexing thread count, so I did a local
> workaround to limit the number of threads in code); I had very good write
> throughput. But with 7.0, I am getting comparable throughput only at
> indexing threadcount > 50. What could be wrong ?
>
>
> Thanks @Erick, I checked the commit settings, both soft and hard commits
> are off.
>
>
>
>
> On Tue, Oct 17, 2017 at 3:47 AM, Amrit Sarkar <sarkaramrit2@gmail.com>
> wrote:
>
>> >
>> > In 7.0, i am finding that the file is written to disk very early on
>> > and it is being updated every second or so. Had something changed in 7.0
>> > which is causing it?  I tried something similar with solr 6.5 and i was
>> > able to get almost a GB size files on disk.
>>
>>
>> Interesting observation, Nawab, with ramBufferSizeMB=20G, you are getting
>> 20GB segments on 6.5 or less? a GB?
>>
>> Amrit Sarkar
>> Search Engineer
>> Lucidworks, Inc.
>> 415-589-9269
>> www.lucidworks.com
>> Twitter http://twitter.com/lucidworks
>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>>
>> On Tue, Oct 17, 2017 at 12:48 PM, Nawab Zada Asad Iqbal <khichi@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I have  tuned  (or tried to tune) my settings to only flush the segment
>> > when it has reached its maximum size. At the moment,I am using my
>> > application with only a couple of threads (i have limited to one thread
>> for
>> > analyzing this scenario) and my ramBufferSizeMB=20000 (i.e. ~20GB). With
>> > this, I assumed that my file sizes on the disk will be at in the order of
>> > GB; and no segments will be flushed until the segment's in memory size is
>> > 2GB. In 7.0, i am finding that the file is written to disk very early on
>> > and it is being updated every second or so. Had something changed in 7.0
>> > which is causing it?  I tried something similar with solr 6.5 and i was
>> > able to get almost a GB size files on disk.
>> >
>> > How can I control it to not write to disk until the segment has reached
>> its
>> > maximum permitted size (1945 MB?) ? My write traffic is 'new only' (i.e.,
>> > it doesn't delete any document) , however I also found following
>> infostream
>> > logs, which incorrectly say 'delete=true':
>> >
>> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-887) [   x:filesearch]
>> > o.a.s.c.S.Request [filesearch]  webapp=/solr path=/update
>> > params={commit=false} status=0 QTime=21
>> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
>> > o.a.s.u.LoggingInfoStream [DW][qtp761960786-889]: anyChanges?
>> > numDocsInRam=4434 deletes=true hasTickets:false
>> pendingChangesInFullFlush:
>> > false
>> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
>> > o.a.s.u.LoggingInfoStream [IW][qtp761960786-889]: nrtIsCurrent:
>> infoVersion
>> > matches: false; DW changes: true; BD changes: false
>> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
>> > o.a.s.c.S.Request [filesearch]  webapp=/solr path=/admin/luke
>> > params={show=index&numTerms=0&wt=json} status=0 QTime=0
>> >
>> >
>> >
>> > Thanks
>> > Nawab
>> >
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message