uima-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Osborne, John D" <josbo...@uabmc.edu>
Subject RE: UIMA analysis from a database
Date Fri, 15 Sep 2017 02:28:14 GMT
Hi Nicolas,

I'm curious, why did you decide to use Spark over UIMA-AS or DUCC? Is it because you are more
familiar with Spark or were their other reasons?

I have been using UIMA-AS, I am currently experimenting with DUCC and would love to hear your
thoughts on the matter.

 -John


________________________________________
From: Nicolas Paris [niparisco@gmail.com]
Sent: Thursday, September 14, 2017 5:32 PM
To: user@uima.apache.org
Subject: Re: UIMA analysis from a database

Hi Benedict

Not sure this is helpful for you, but only an advice.
I recommend usint UIMA for what it is first intended : nlp pipeline.

When dealing with multi threaded application, I would go for dedicated
technologies.

I have been successfuly using UIMA together with apache spark. While this
design works well on a single computer, I am now able to distribute UIMA
pipeline over dosen of them, withou extra need.

Then I focus on UIMA pipeline doing it's job well, and after testing,
industrialize them over spark.

Advantages of this design are:
- benefit from spark distributing expertise (note failure, memory
  consumption, data partitionning...)
- simplify UIMA programming (no multithread inside, only NLP stuff)
- scale when needed (add more chip computer, get better performances)
- get expertise with spark, and use it with any java code you d'like
- spark do have JDBC connectors and may be able to fetch data in
  multithread easily.

you can have an wotking example in my repo https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_parisni_UimaOnSpark&d=DwIDAw&c=o3PTkfaYAd6-No7SurnLtwPssd47t-De9Do23lQNz7U&r=SEpLmXf_P21h_X0qEQSssKMDDEOsGxxYoSxofi_ZbFo&m=9w0-CmPPbyYElPML1EOD_jqp84ZXz2rpRpEFsYxecTY&s=RQJDDNPq5uLPH4q0rY6tvPy_CxvFLjUCkLqpnPCeSgU&e=
This have not been simple to make it working, but I can tell know this
methods is robust and optimized.


Le 14 sept. 2017 à 21:24, Benedict Holland écrivait :
> Hello everyone,
>
> I am trying to get my project off the ground and hit a small problem.
>
> I want to read text from a large database (lets say, 100,000+ rows). Each
> row will have a text article. I want to connect to the database, request a
> single row from the database, and process this document through an NLP
> engine and I want to do this in parallel. Each document will be say, split
> up into sentences and each sentence will be POS tagged.
>
> After reading the documentation, I am more confused than when I started. I
> think I want something like the FileSystemCollectionReader example and
> build a CPE. Instead of reading from the file system, it will read from the
> database.
>
> There are two problems with this approach:
>
> 1. I am not sure it is multi threaded: CAS initializers are deprecated and
> it appears that the getNext() method will only run in a single thread.
> 2. The FileSystemCollectionReader loads references to the file location
> into memory but not the text itself.
>
> For problem 1, the line I find very troubling is
>
> File file = (File) mFiles.get(mCurrentIndex++);
>
> I have to assume from this line that the CollectionReader_ImplBase is not
> multi-threaded but is intended to rapidly iterate over a set of documents
> in a single thread.
>
> Problem 2 is easily solved as I can create a massive array of integers if I
> feel like.
>
> Anyway, after deciding that this is not likely the solution, I looked into
> Multi-view Sofa annotators. I don't think these do what I want either. In
> this context, I would treat the database table as a single object with many
> "views" being chunks of rows. I don't think this works, based on the
> SofaExampleAnnotator code provided. It also appears to run in a single
> thread.
>
> This leaves me with CAS pools. I know that this is doing to be
> multi-threaded. I believe I create however many CAS objects from the
> annotator I want, probably an aggregate annotator. Is this correct and am I
> on the right track with CAS Pools?
>
> Thank you so much,
> ~Ben

Mime
View raw message