Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of noble.paul@gmail.com
 designates 74.125.78.25 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=lo+ojOHLdbKQkNCKQgimYDY5mnAt2Ue9+lxcS2hDLMaDFUOiy0h8A4HDZK1ZbP9hIW
         /MUqPA+hEhvdTIwNHbskuCvd5iS2/3HMD1Ll8dqjg33ClhlCFBcDCYJWGfAexPrAmYFO
         1nt6eWuVJ6MVWM6+BY8dF2Lumi7Jb4e8nkV4s=
Message-ID: <5e76b0ad0811032033n2e0bffb7vc0d3ba92f1d297a1@mail.gmail.com>
Date: Tue, 4 Nov 2008 10:03:11 +0530
From: "=?UTF-8?B?Tm9ibGUgUGF1bCDgtKg=?=
 =?UTF-8?B?4LWL4LSs4LS/4LSz4LWN4oCNIOCkqOCli+CkrOCljeCks+CljQ==?="
 <noble.paul@gmail.com>
To: solr-user@lucene.apache.org
Subject: Re: MySql / Solr 1.3 / Tomcat55 - Full Import for 8,5M of data >>
 Exception in thread "Thread-33"
In-Reply-To: <20305986.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <20305986.post@talk.nabble.com>

>From the data-config.xml it is obvious that the your indexing will
take a lot of time. MySql has very poor join performance. It is not a
very good idea to run this on a production database.

I would suggest you to configure another mysql server and do mysql
replication to that and run the import from there.


On Mon, Nov 3, 2008 at 10:06 PM, sunnyfr <johanna.34@gmail.com> wrote:
>
> Hi,
>
> I've put a batchsize parameter at -1, it works fine, the point is I will
> monopolize the MySql's database for 10hours.
> And other request on it like update, or other process will be stack. And if
> I don't use batchsize -1 I will have an OOM error like below. I tried to put
> batchsize 1000 or 1 but it doesn't work either.
>
> What would you reckon ?
> I can't have another server just for that, I have to use MySql production's
> database.
>
>> Should I try to generate CSV files from MySql and then make a full import
>> from it, and delta import on the database ??
>
>> Should I modify the connection code in solr, to try to manage this memory.
>
>> Do you have another idea ?
>
> If anyone has any suggestions/needs more info, i'd be very greatful.
>
>
>
> FYI : Linux server / 8G Mem
>
> TomCat55 :
> JAVA_OPTS="-Xms2000m -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError
> -XX:+UseParallelGC -XX:+AggressiveOpts -XX:NewRatio=5 -Xloggc:gc.log
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>
> data-config.xml:
> http://www.nabble.com/file/p20305986/data-config.xml data-config.xml
>
>
> My Error:
> Nov  3 16:45:43 solr-test jsvc.exec[29099]:  [PSYoungGen:
> 227583K->0K(455104K)] [PSOldGen: 3413375K->142574K(1706688K)]
> 3640959K->142574K(2161792K)
> Nov  3 16:45:43 solr-test jsvc.exec[29099]:  [PSPermGen:
> 20751K->20751K(21504K)], 1.0853010 secs] [Times: user=0.99 sys=0.10,
> real=1.09 secs]
> Nov  3 16:45:43 solr-test jsvc.exec[29099]: Nov 3, 2008 4:45:43 PM
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
> processChildren SEVERE: Exception invoking periodic operation:
> java.lang.OutOfMemoryError: Java heap space ^Iat
> sun.nio.cs.US_ASCII.newDecoder(US_ASCII.java:39) ^Iat
> java.nio.charset.CharsetEncoder.isLegalReplacement(CharsetEncoder.java:311)
> ^Iat java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:267)
> ^Iat java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:186) ^Iat
> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:209) ^Iat
> sun.nio.cs.US_ASCII$Encoder.<init>(US_ASCII.java:121) ^Iat
> sun.nio.cs.US_ASCII$Encoder.<init>(US_ASCII.java:118) ^Iat
> sun.nio.cs.US_ASCII.newEncoder(US_ASCII.java:43) ^Iat
> java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215) ^Iat
> java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207) ^Iat
> java.lang.StringCoding.encode(StringCoding.java:266) ^Iat
> java.lang.String.getBytes(String.java:947) ^Iat java.io.UnixFi
> Nov  3 16:45:43 solr-test jsvc.exec[29099]:
> java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228) ^Iat
> java.io.File.isDirectory(File.java:754) ^Iat
> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:873)
> ^Iat org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
> ^Iat org.apache.catalina.startup.HostConfig.check(HostConfig.java:1206) ^Iat
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:293)
> ^Iat
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
> ^Iat
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1306)
> ^Iat
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1570)
> ^Iat
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1579)
> ^Iat
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1559)
> ^Iat java.lang.Thread.run(Thread.java:619)
>
> --
> View this message in context: http://www.nabble.com/MySql---Solr-1.3---Tomcat55-----Full-Import-for-8%2C5M-of-data-%3E%3E-Exception-in-thread-%22Thread-33%22-tp20305986p20305986.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


-- 
--Noble Paul