Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: Re: Cache problem
To: solr-user@lucene.apache.org
References: <56FD255D.8070803@mdpi.com> <570BA902.9040201@mdpi.com>
 <EE623384-D85C-43E8-A320-60277CC90632@gmail.com>
 <CAGyT8SXTLbZCKX8+iL5nG44YoF2cUB=PANO7eNzs1mZayi21sA@mail.gmail.com>
 <570CC14C.4040906@mdpi.com> <570D9716.6040504@elyograg.org>
 <570DEDCC.6030106@mdpi.com> <570DFB2E.8080408@elyograg.org>
 <570E20C7.3040101@mdpi.com>
From: Shawn Heisey <apache@elyograg.org>
Message-ID: <570E42B2.2030702@elyograg.org>
Date: Wed, 13 Apr 2016 06:59:30 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.7.2
MIME-Version: 1.0
In-Reply-To: <570E20C7.3040101@mdpi.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

On 4/13/2016 4:34 AM, Bastien Latard - MDPI AG wrote:
> Thank you all again for your good and detailed answer.
> I will combine all of them to try to build a better environment.
>
> *Just a last question...*
> /I don't remember exactly when I needed to increase the java heap.../
> /but is it possible that this was for the DataImport.../
>
> *Would the DIH work if it cannot "load" the temporary index into the
> java heap in the full-index mode?*
> I thought that's why I needed to increase this value...but I might be
> confused!

The default behavior on many JDBC drivers is to load the *entire* SQL
result into memory *before* sending those results to the requesting
application.  This is the way the MySQL driver behaves by default, and
the way that older versions of the Microsoft driver for SQL Server
behave by default.

There should be a way to tell the JDBC driver to stream the results back
instead of loading them into memory.  For MySQL, you just have to set
the batchSize parameter in the DIH config to -1, which causes the
underlying code to do "setFetchSize(Integer.MIN_VALUE)".  For SQL
Server, you need a recent version of the driver, where they changed the
default behavior.   For other databases, you may need a JDBC url parameter.

Thanks,
Shawn