Return-Path: 
 <dev-return-94365-apmail-lucene-dev-archive=lucene.apache.org@lucene.apache.org>
X-Original-To: apmail-lucene-dev-archive@www.apache.org
Delivered-To: apmail-lucene-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD12E94E1
	for <apmail-lucene-dev-archive@www.apache.org>;
 Fri, 30 Mar 2012 13:02:51 +0000 (UTC)
Received: (qmail 21345 invoked by uid 500); 30 Mar 2012 13:02:50 -0000
Delivered-To: apmail-lucene-dev-archive@lucene.apache.org
Received: (qmail 21271 invoked by uid 500); 30 Mar 2012 13:02:50 -0000
Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:dev@lucene.apache.org>
List-Id: <dev.lucene.apache.org>
Reply-To: dev@lucene.apache.org
Delivered-To: mailing list dev@lucene.apache.org
Received: (qmail 21264 invoked by uid 99); 30 Mar 2012 13:02:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Mar 2012 13:02:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Mar 2012 13:02:47 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 88E9F34E599
	for <dev@lucene.apache.org>; Fri, 30 Mar 2012 13:02:26 +0000 (UTC)
Date: Fri, 30 Mar 2012 13:02:26 +0000 (UTC)
From: "Bernd Fehling (Commented) (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <1338056566.37661.1333112546566.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1410407962.18208.1325938899242.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (SOLR-3011) DIH MultiThreaded bug
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/SOLR-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242310#comment-13242310 ] 

Bernd Fehling commented on SOLR-3011:
-------------------------------------

Just tried multi-threaded. It produces the required number of threads (seen in debugger) but only runs once.
My configuration is:
<dataConfig>
        <dataSource name="filetraverser" type="FileDataSource" encoding="UTF-8" />
        <document>
        <entity name="basedata" processor="FileListEntityProcessor"
                threads="4"
                rootEntity="false"
                fileName="\.xml$"
                recursive="true"
                dataSource="null"
                baseDir="/srv/www/solr/DATA/OAI"
                >
                <entity name="records" processor="XPathEntityProcessor"
                        threads="4"
                        rootEntity="true"
                        dataSource="filetraverser"
                        stream="true"
                        forEach="/documents/document"
                        url="${basedata.fileAbsolutePath}"
                        >
                        <field column="id"      xpath="/documents/document/@id" />
                        <field column="dctitle" xpath="/documents/document/element[@name='dctitle']/value" />
                </entity>
        </entity>
        </document>
</dataConfig>

It should read all files below baseDir and build documents from the records inside the files.
Works fine in non-multi-threaded but only reads the first file in multi-threaded mode.
Any idea?


And another thing to mention, in TestThreaded.java there are the lines:
  @Test
  public void testCachedThreadless_FullImport() throws Exception {
    runFullImport(getCachedConfig(random.nextBoolean(), random.nextBoolean(), 0));
  }
  
  @Test
  public void testCachedSingleThread_FullImport() throws Exception {
    runFullImport(getCachedConfig(random.nextBoolean(), random.nextBoolean(), 1));
  }
  
  @Test
  public void testCachedThread_FullImport() throws Exception {
    int numThreads = random.nextInt(9) + 1; // between one and 10
    String config = getCachedConfig(random.nextBoolean(), random.nextBoolean(), numThreads);
    runFullImport(config);
  }

This will test 0, 1 and random between 1 to 9. But 1 is already covered.
So wouldn't it be better to have "random.nextInt(8) + 2" for the range 2 to 9?

                
> DIH MultiThreaded bug
> ---------------------
>
>                 Key: SOLR-3011
>                 URL: https://issues.apache.org/jira/browse/SOLR-3011
>             Project: Solr
>          Issue Type: Sub-task
>          Components: contrib - DataImportHandler
>    Affects Versions: 3.5
>            Reporter: Mikhail Khludnev
>            Assignee: James Dyer
>            Priority: Minor
>             Fix For: 3.6
>
>         Attachments: SOLR-3011.patch, SOLR-3011.patch, SOLR-3011.patch, SOLR-3011.patch, SOLR-3011.patch, patch-3011-EntityProcessorBase-iterator.patch, patch-3011-EntityProcessorBase-iterator.patch
>
>
> current DIH design is not thread safe. see last comments at SOLR-2382 and SOLR-2947. I'm going to provide the patch makes DIH core threadsafe. Mostly it's a SOLR-2947 patch from 28th Dec. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org