Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
MIME-Version: 1.0
In-Reply-To: <0994d1d7-31d5-c2cd-30a4-5ba55b5390f3@leirtech.com>
References: <CABxSQx+9Ad0Wk0YuYgmUJ1h_TaL1Pxu4k1rvjCbt+AfO3K6MCQ@mail.gmail.com>
 <SN1PR09MB1069FCF46C1BB704DA7E5C73C7C20@SN1PR09MB1069.namprd09.prod.outlook.com>
 <CAJuqOKJVUb0hGbuYOT+3Jn+PJM16_s8hFCtg3qUyg-5oSeaytg@mail.gmail.com>
 <D9523EC2-258D-48D4-8577-BD84CF2ABC7F@wunderwood.org> <8989cd1b-d4e6-045a-c307-c8e1f9d33dd0@leirtech.com>
 <40C176A0-63D5-4A18-BF9D-F54358537EB4@wunderwood.org> <0994d1d7-31d5-c2cd-30a4-5ba55b5390f3@leirtech.com>
From: Erick Erickson <erickerickson@gmail.com>
Date: Thu, 6 Oct 2016 08:16:49 -0700
Message-ID: <CAN4YXvfBOU8_7m5pz5LYBq1-z7d_zS3zbyxssS46G3QHzwzyiA@mail.gmail.com>
Subject: Re: SOLR Sizing
To: solr-user <solr-user@lucene.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 06 Oct 2016 15:17:39 -0000

OCR _without errors_ wouldn't break it. That comment assumed that the OCR
was dirty I thought.

Honest, I once was trying to index an OCR'd image of a "family tree" that w=
as a
stylized tree where the most remote ancestor was labeled in vertical text o=
n the
trunk, and descendants at various angles as the trunk branched, the branche=
s
branched and on and on....

And as far as cleaning up the text is concerned if it's dirty,
anything you do is
wrong. For instance, again using the genealogy example, throwing out
unrecognized
words like, removes the data that's important when they're names.

But leaving nonsense characters in is wrong too....

And hand-correcting all of the data is almost always far too expensive.

If your OCR is, indeed perfect, then I envy you ;)...

On a different note, I thought the captcha-image way of correcting OCR
text was brilliant.

Erick

On Thu, Oct 6, 2016 at 8:05 AM, Rick Leir <rleir@leirtech.com> wrote:
> I am curious to know where the square-root assumption is from, and why OC=
R
> (without errors) would break it. TIA
>
> cheers - - Rick
>
> On 2016-10-04 10:51 AM, Walter Underwood wrote:
>>
>> No, we don=E2=80=99t have OCR=E2=80=99ed text. But if you do, it breaks =
the assumption
>> that vocabulary size
>> is the square root of the text size.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>>
>>
>>> On Oct 4, 2016, at 7:14 AM, Rick Leir <rleir@leirtech.com> wrote:
>>>
>>> OCR=E2=80=99ed text can have large amounts of garbage such as '';,-d'."
>>> particularly when there is poor image quality or embedded graphics. Is =
that
>>> what is causing your huge vocabularies? I filtered the text, removing a=
ny
>>> word with fewer than 3 alphanumerics or more than 2 non-alphas.
>>>
>>>
>>> On 2016-10-03 09:30 PM, Walter Underwood wrote:
>>>>
>>>> That approach doesn=E2=80=99t work very well for estimates.
>>>>
>>>> Some parts of the index size and speed scale with the vocabulary inste=
ad
>>>> of the number of documents.
>>>> Vocabulary usually grows at about the square root of the total amount =
of
>>>> text in the index. OCR=E2=80=99ed text
>>>> breaks that estimate badly, with huge vocabularies.
>>>>
>>>>
>