Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Subject: Re: problems with bulk indexing with concurrent DIH
To: solr-user@lucene.apache.org
References: <65db1341-f46a-7ea3-4ea9-a8c706b01a2e@uni-bielefeld.de>
 <28034_1469628271_u6RE4UWi019021_CAN4YXvdsAE7yiFuNCWThU4nJoi+BD7OS4YiwqWnat=3M7knb8Q@mail.gmail.com>
 <c2fa382a-a356-cd45-e6ee-f5a54f59d8bc@uni-bielefeld.de>
 <10932_1469630487_u6REfQWp011793_CAN4YXveBm_ufYpaZEmz0FOxhZdK3NT065VuUDg7E9+Mn_CwkeA@mail.gmail.com>
 <632c67cc-6570-dde3-846f-536cc441755d@uni-bielefeld.de>
 <CAF8TkC5T432pLA6y3L2ENJPRQB-vO+ndsvSypJvS=26f2PhgZQ@mail.gmail.com>
 <3ae0d341-37ed-596f-ccd8-3d1269846ca4@uni-bielefeld.de>
 <CAF8TkC53y+azWWrE84mXfDn+Yu=W1MAnCzm=5r0qAG9dcLm+Dg@mail.gmail.com>
From: Bernd Fehling <bernd.fehling@uni-bielefeld.de>
Message-id: <64769e20-89a9-632c-d61b-5798ed78c8df@uni-bielefeld.de>
Date: Thu, 04 Aug 2016 09:31:24 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.2
In-reply-to: 
 <CAF8TkC53y+azWWrE84mXfDn+Yu=W1MAnCzm=5r0qAG9dcLm+Dg@mail.gmail.com>
archived-at: Thu, 04 Aug 2016 07:31:44 -0000

After updating to version 5.5.3 it looks good now.
I think LUCENE-6161 has fixed my problem.
Nevertheless, after updating my development system and recompyling
my plugins I will have a look at DIH about the "update" and also
your advise about the uniqueKey.

Best regards
Bernd

Am 02.08.2016 um 16:16 schrieb Mikhail Khludnev:
> These deletes seem really puzzling to me. Can you experiment with
> commenting uniqeKey in schema.xml. My expectation that deletes should go
> away after that.
> 
> On Tue, Aug 2, 2016 at 4:50 PM, Bernd Fehling <
> bernd.fehling@uni-bielefeld.de> wrote:
> 
>> Hi Mikhail,
>>
>> there are no deletes at all from my point of view.
>> All records have unique id's.
>> No sharding at all, it is a single index and it is certified
>> that all DIH's get different data to load and no record is
>> sent twice to any DIH participating at concurrent loading.
>>
>> Only assumption so far, DIH is sending the records as "update"
>> (and not pure "add") to the indexer which will generate delete
>> files during merge. If the number of segments is high it will
>> take quite long to merge and check all records of all segments.
>>
>> I'm currently setting up SOLR 5.5.3 but that takes a while.
>> I also located an "overwrite" parameter somewhere in DIH which
>> will force an "add" and not an "update" to the index, but
>> couldn't figure out how to set the parameter with command.
>>
>> Bernd
>>
>>
>> Am 02.08.2016 um 15:15 schrieb Mikhail Khludnev:
>>> Bernd,
>>> But why do you have so many deletes? Is it expected?
>>> When you run DIHs concurrently, do you shard intput data by uniqueKey?
>>>
>>> On Wed, Jul 27, 2016 at 6:20 PM, Bernd Fehling <
>>> bernd.fehling@uni-bielefeld.de> wrote:
>>>
>>>> If there is a problem in single index then it might also be in
>> CloudSolr.
>>>> As far as I could figure out from INFOSTREAM, documents are added to
>>>> segments
>>>> and terms are "collected". Duplicate term are "deleted" (or whatever).
>>>> These deletes (or whatever) are not concurrent.
>>>> I have a lines like:
>>>> BD 0 [Wed Jul 27 13:28:48 GMT+01:00 2016; Thread-27879]: applyDeletes:
>>>> infos=...
>>>> BD 0 [Wed Jul 27 13:31:48 GMT+01:00 2016; Thread-27879]: applyDeletes
>> took
>>>> 180028 msec
>>>> ...
>>>> BD 0 [Wed Jul 27 13:42:03 GMT+01:00 2016; Thread-27890]: applyDeletes:
>>>> infos=...
>>>> BD 0 [Wed Jul 27 14:38:55 GMT+01:00 2016; Thread-27890]: applyDeletes
>> took
>>>> 3411845 msec
>>>>
>>>> 3411545 msec are about 56 minutes where the system is doing what???
>>>> At least not indexing because only one JAVA process and no I/O at all!
>>>>
>>>> How can SolrJ help me now with this problem?
>>>>
>>>> Best
>>>> Bernd
>>>>
>>>>
>>>> Am 27.07.2016 um 16:41 schrieb Erick Erickson:
>>>>> Well, at least it'll be easier to debug in my experience. Simple
>> example.
>>>>> At some point you'll call CloudSolrClient.add(doc list). Comment just
>>>> that
>>>>> out and you'll be able to isolate whether the issue is querying the be
>> or
>>>>> sending to Solr.
>>>>>
>>>>> Then CloudSolrClient (assuming SolrCloud) has efficiencies in terms of
>>>>> routing...
>>>>>
>>>>> Best
>>>>> Erick
>>>>>
>>>>> On Jul 27, 2016 7:24 AM, "Bernd Fehling" <
>> bernd.fehling@uni-bielefeld.de
>>>>>
>>>>> wrote:
>>>>>
>>>>>> So writing some SolrJ doing the same job as the DIH script
>>>>>> and using that concurrent will solve my problem?
>>>>>> I'm not using Tika.
>>>>>>
>>>>>> I don't think that DIH is my problem, even if it is not the best
>>>> solution
>>>>>> right now.
>>>>>> Nevertheless, you are right SolrJ has higher performance, but what
>>>>>> if I have the same problems with SolrJ like with DIH?
>>>>>>
>>>>>> If it runs with DIH it should run with SolrJ with additional
>> performance
>>>>>> boost.
>>>>>>
>>>>>> Bernd
>>>>>>
>>>>>>
>>>>>> On 27.07.2016 at 16:03, Erick Erickson:
>>>>>>> I'd actually recommend you move to a SolrJ solution
>>>>>>> or similar. Currently, you're putting a load on the Solr
>>>>>>> servers (especially if you're also using Tika) in addition
>>>>>>> to all indexing etc.
>>>>>>>
>>>>>>> Here's a sample:
>>>>>>> https://lucidworks.com/blog/2012/02/14/indexing-with-solrj/
>>>>>>>
>>>>>>> Dodging the question I know, but DIH sometimes isn't
>>>>>>> the best solution.
>>>>>>>
>>>>>>> Best,
>>>>>>> Erick
>>>>>>>
>>>>>>> On Wed, Jul 27, 2016 at 6:59 AM, Bernd Fehling
>>>>>>> <bernd.fehling@uni-bielefeld.de> wrote:
>>>>>>>> After enhancing the server with SSDs I'm trying to speed up
>> indexing.
>>>>>>>>
>>>>>>>> The server has 16 CPUs and more than 100G RAM.
>>>>>>>> JAVA (1.8.0_92) has 24G.
>>>>>>>> SOLR is 4.10.4.
>>>>>>>> Plain XML data to load is 218G with about 96M records.
>>>>>>>> This will result in a single index of 299G.
>>>>>>>>
>>>>>>>> I tried with 4, 8, 12 and 16 concurrent DIHs.
>>>>>>>> 16 and 12 was to much because for 16 CPUs and my test continued
>> with 8
>>>>>> concurrent DIHs.
>>>>>>>> Then i was trying different <indexConfig> and <updateHandler>
>> settings
>>>>>> but now I'm stuck.
>>>>>>>> I can't figure out what is the best setting for bulk indexing.
>>>>>>>> What I see is that the indexing is "falling asleep" after some time
>> of
>>>>>> indexing.
>>>>>>>> It is only producing del-files, like _11_1.del, _w_2.del,
>> _h_3.del,...
>>>>>>>>
>>>>>>>> <indexConfig>
>>>>>>>>     <maxIndexingThreads>8</maxIndexingThreads>
>>>>>>>>     <ramBufferSizeMB>1024</ramBufferSizeMB>
>>>>>>>>     <maxBufferedDocs>-1</maxBufferedDocs>
>>>>>>>>     <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>>>>>>>       <int name="maxMergeAtOnce">8</int>
>>>>>>>>       <int name="segmentsPerTier">100</int>
>>>>>>>>       <int name="maxMergedSegmentMB">512</int>
>>>>>>>>     </mergePolicy>
>>>>>>>>     <mergeFactor>8</mergeFactor>
>>>>>>>>     <mergeScheduler
>>>>>> class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
>>>>>>>>     <lockType>${solr.lock.type:native}</lockType>
>>>>>>>>     ...
>>>>>>>> </indexConfig>
>>>>>>>>
>>>>>>>> <updateHandler class="solr.DirectUpdateHandler2">
>>>>>>>>      ### no autocommit at all
>>>>>>>>      <autoSoftCommit>
>>>>>>>>        <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
>>>>>>>>      </autoSoftCommit>
>>>>>>>> </updateHandler>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> command=full-import&optimize=false&clean=false&commit=false&waitSearcher=false
>>>>>>>> After indexing finishes there is a final optimize.
>>>>>>>>
>>>>>>>> My idea is, if 8 DIHs use 8 CPUs then I have 8 CPUs left for merging
>>>>>>>> (maxIndexingThreads/maxMergeAtOnce/mergeFactor).
>>>>>>>> It should do no commit, no optimize.
>>>>>>>> ramBufferSizeMB is high because I have plenty of RAM and I want make
>>>>>> use the speed of RAM.
>>>>>>>> segmentsPerTier is high to reduce merging.
>>>>>>>>
>>>>>>>> But somewhere is a misconfiguration because indexing gets stalled.
>>>>>>>>
>>>>>>>> Any idea what's going wrong?
>>>>>>>>
>>>>>>>>
>>>>>>>> Bernd
>>>>>>>>
>>>>>>
>>>>>