Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jules@limitless.co.uk
 designates 85.119.248.221 as permitted sender)
Message-ID: <5514A034.1090201@limitless.co.uk>
Date: Fri, 27 Mar 2015 13:11:32 +1300
From: Julian Perry <jules@limitless.co.uk>
Organization: Limitless Internet Solutions
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10;
 rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: solr-user@lucene.apache.org
Subject: Re: Build index from Oracle, adding fields
References: <5514940B.2050608@limitless.co.uk> <5514994D.3030306@elyograg.org>
In-Reply-To: <5514994D.3030306@elyograg.org>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

On 27/03/2015 12:42, Shawn Heisey wrote:
> If that's not practical, then the only real option you have is to drop
> back to one entity, and build a single SELECT statement (using JOIN and
> some form of CONCAT) that will gather all the information from all the
> tables at the same time, and combine multiple values together into one
> SQL result field with some kind of delimiter.  Then you can use the
> RegexTransformer's "splitBy" functionality to turn the concatenated data
> back into multiple values for your multi-valued field.  Database servers
> tend to be REALLY good at JOIN operations, so the database would be
> doing the heavy lifting.

I did try that in fact (and do it with one of my other indexes).

However, with this index the sub-select can return 200 rows of
200 characters - and that blows up in Oracle as the field is
over 4000 characters long (and the work-around for that is to
use clob's - but that has its own performance problems).

Currently I am doing this by exporting a CSV file and
processing it with a C program - and then reading the CSV with
SOLR :(

-- 
Cheers
Jules.