uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Eckart de Castilho <...@apache.org>
Subject Re: Working with very large text documents
Date Fri, 18 Oct 2013 10:38:11 GMT
Well, assuming this would e.g. be a server log, you could want to notice that some IP or set
of IPs tried to log in with different user accounts across an extended period of time. So
even if there is no linguistic relationship here, there is definitely a relationship that
a security person would want to be able to discover. But that may be a secondary step after
parsing the individual log lines.

-- Richard

On 18.10.2013, at 12:34, Jens Grivolla <j+asf@grivolla.net> wrote:

> Ok, but then log files are usually very easy to split since they normally consist of
independent lines. So you could just have one document per day or whatever gets it down to
a reasonable size, without the risk of breaking grammatical or semantic relationships.
> On 10/18/2013 12:25 PM, Armin Wegner wrote:
>> Hi Jens,
>> It's a log file.
>> Cheers,
>> Armin
>> -----Urspr√ľngliche Nachricht-----
>> Von: Jens Grivolla [mailto:j+asf@grivolla.net]
>> Gesendet: Freitag, 18. Oktober 2013 11:05
>> An: user@uima.apache.org
>> Betreff: Re: Working with very large text documents
>> On 10/18/2013 10:06 AM, Armin Wegner wrote:
>>> What are you doing with very large text documents in an UIMA Pipeline, for example
9 GB in size.
>> Just out of curiosity, how can you possibly have 9GB of text that represent one document?
From a quick look at project gutenberg it seems that a full book with HTML markup is about
500kB to 1MB, so that's about a complete public library full of books.
>> Bye,
>> Jens

View raw message