Return-Path: 
 <dev-return-97716-apmail-lucene-dev-archive=lucene.apache.org@lucene.apache.org>
X-Original-To: apmail-lucene-dev-archive@www.apache.org
Delivered-To: apmail-lucene-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 52BC39BBC
	for <apmail-lucene-dev-archive@www.apache.org>;
 Mon,  7 May 2012 19:29:15 +0000 (UTC)
Received: (qmail 30624 invoked by uid 500); 7 May 2012 19:29:14 -0000
Delivered-To: apmail-lucene-dev-archive@lucene.apache.org
Received: (qmail 30554 invoked by uid 500); 7 May 2012 19:29:14 -0000
Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:dev@lucene.apache.org>
List-Id: <dev.lucene.apache.org>
Reply-To: dev@lucene.apache.org
Delivered-To: mailing list dev@lucene.apache.org
Received: (qmail 30546 invoked by uid 99); 7 May 2012 19:29:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 May 2012 19:29:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 May 2012 19:29:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9EBE94388BC
	for <dev@lucene.apache.org>; Mon,  7 May 2012 19:28:49 +0000 (UTC)
Date: Mon, 7 May 2012 19:28:49 +0000 (UTC)
From: "Mikhail Khludnev (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <1097216975.36113.1336418929651.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <76973248.28785.1334589378651.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (SOLR-3360) Problem with DataImportHandler
 multi-threaded
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/SOLR-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269925#comment-13269925 ] 

Mikhail Khludnev commented on SOLR-3360:
----------------------------------------

James,

I can't reproduce the failure. 
mkhl$ ant test-contrib -Dtests.seed=-55eeb72d0a16dfec:4e1a59f5738a6b25:4bf3cbf2bd3b659a -Dtestcase=TestThreaded

junit report 

{code}
<property name="tests.seed" value="-55eeb72d0a16dfec:4e1a59f5738a6b25:4bf3cbf2bd3b659a" />

  <testcase classname="org.apache.solr.handler.dataimport.TestThreaded" name="testCachedThread_FullImport" time="0.965" />
  <testcase classname="org.apache.solr.handler.dataimport.TestThreaded" name="testCachedThreadless_FullImport" time="0.055" />
  <testcase classname="org.apache.solr.handler.dataimport.TestThreaded" name="testCachedSingleThread_FullImport" time="0.053" />

{code}


Finally I added _Total test which enumerates all test params 
https://github.com/m-khl/solr-patches/commit/0532e653a3319247519f90bd8987c84171ac6a56.diff

at core i5 MacBook Pro

:solr mkhl$ ant test-contrib -Dtestcase=TestThreaded
junit-sequential:
    [junit] Testsuite: org.apache.solr.handler.dataimport.TestThreaded
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 3.899 sec
    [junit] 

junit-parallel:

{code}
 </properties>
  <testcase classname="org.apache.solr.handler.dataimport.TestThreaded" name="testCachedThread_FullImport" time="1.042" />
  <testcase classname="org.apache.solr.handler.dataimport.TestThreaded" name="testCachedThreadless_FullImport" time="0.047" />
  <testcase classname="org.apache.solr.handler.dataimport.TestThreaded" name="testCachedSingleThread_FullImport" time="0.052" />
  <testcase classname="org.apache.solr.handler.dataimport.TestThreaded" name="testCachedThread_Total" time="0.898" />
  <system-out><![CDATA[]]></system-out>
{code}


Please give me a clue how to reproduce the failure. What do you use IDE or script? Have you clean before test? Pls show me exact command, junit report, log/output, etc
                
> Problem with DataImportHandler multi-threaded
> ---------------------------------------------
>
>                 Key: SOLR-3360
>                 URL: https://issues.apache.org/jira/browse/SOLR-3360
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 3.6
>         Environment: Solr 3.6.0, Apache Tomcat 6.0.20, jdk1.6.0_15, Windows XP
>            Reporter: Claudio R
>            Assignee: James Dyer
>             Fix For: 3.6.1
>
>         Attachments: SOLR-3360-test.patch, SOLR-3360-test.patch, SOLR-3360-test.patch, SOLR-3360-test.patch, SOLR-3360-test.patch, SOLR-3360-test.patch, SOLR-3360.patch
>
>
> Hi,
> If I use dataimport with 1 thread, I got:
> <lst name="statusMessages">
>    <str name="Total Requests made to DataSource">5001</str>
>    <str name="Total Rows Fetched">1000</str>
>    <str name="Total Documents Skipped">0</str>
>    <str name="Full Dump Started">2012-04-16 11:21:57</str>
>    <str name="">Indexing completed. Added/Updated: 1000 documents. Deleted 0 documents.</str>
>    <str name="Committed">2012-04-16 11:23:19</str>
>    <str name="Total Documents Processed">1000</str>
>    <str name="Time taken">0:1:22.390</str>
> </lst>
> If I use datamport with 10 threads, I got:
> <lst name="statusMessages">
>    <str name="Total Requests made to DataSource">0</str>
>    <str name="Total Rows Fetched">10000</str>
>    <str name="Total Documents Skipped">0</str>
>    <str name="Full Dump Started">2012-04-16 11:31:43</str>
>    <str name="">Indexing completed. Added/Updated: 10000 documents. Deleted 0 documents.</str>
>    <str name="Committed">2012-04-16 11:41:50</str>
>    <str name="Total Documents Processed">10000</str>
>    <str name="Time taken">0:10:7.586</str>
> </lst>
> The configuration of 10 threads consumed 10 times longer than the configuration with 1 thread.
> I have 1000 records in the database.
> My db-data-config.xml is shown below:
> <?xml version="1.0" encoding="UTF-8" ?>
> <dataConfig>
>    <dataSource driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" url="jdbc:sqlserver://200.XXX.XXX.XXX:1433;databaseName=test" user="user" password="pass"/>
>       <document>
>          <entity name="indice" rootEntity="true" threads="10" transformer="RegexTransformer,TemplateTransformer" query="select top 1000 i.id_indice, i.a, i.b from indice i where i.status = 'I'" deltaImportQuery="i.id_indice, i.a, i.b from indice i where id_indice in ('${dataimporter.delta.id_indice}')" deltaQuery="select id_indice from indice where status='I' and data_hora_modificacao >= convert(datetime, '${dataimporter.last_index_time}', 120)" deletedPkQuery="select id_indice from indice where status='D' and data_hora_modificacao >= convert(datetime, '${dataimporter.last_index_time}', 120)">	
>             <field column="id_indice" name="id_indice" />
>             <field column="a" name="a" />
>             <field column="b" name="b" />
>             <entity name="filtro" transformer="RegexTransformer,TemplateTransformer" query="select categoria, sub_categoria from filtro where indice_id_indice = '${indice.id_indice}'">
>                <field name="filtro_categoria" column="categoria" />
>                <field name="filtro_sub_categoria" column="sub_categoria" />
>                <field name="nv_sub_categoria" column="nv_sub_categoria" template="${filtro.categoria}|${filtro.sub_categoria}" />
>             </entity>
>             <entity name="pagina_relacionada" query="select url from pagina_relacionada where indice_id_indice = '${indice.id_indice}'">
>                <field name="pagina_relacionada_url" column="url" />
>             </entity>
>             <entity name="veja_mais" query="select chamada, url from veja_mais where indice_id_indice = '${indice.id_indice}'">
>                <field name="veja_mais_chamada" column="chamada" />
>                <field name="veja_mais_url" column="url" />
>             </entity>
>             <entity name="video" query="select url from video where indice_id_indice = '${indice.id_indice}'">
>                <field name="video_url" column="url" />
>             </entity>
>             <entity name="galeria" query="select url from galeria where indice_id_indice = '${indice.id_indice}'">
>                <field name="galeria_url" column="url" />
>             </entity>
>          </entity>
>       </document>
> </dataConfig>
> Thanks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org