Mailing-List: contact dev-help@ctakes.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ctakes.apache.org
Received-SPF: pass (athena.apache.org: domain of
 Sean.Finan@childrens.harvard.edu designates 134.174.13.91 as permitted
 sender)
From: "Finan, Sean" <Sean.Finan@childrens.harvard.edu>
To: "dev@ctakes.apache.org" <dev@ctakes.apache.org>
Subject: RE: files vs strings in collection reader
Thread-Topic: files vs strings in collection reader
Thread-Index: AQHOS1ewfw7d7nKQ5kidmb4fTZafS5j6XYaA///AlXA=
Date: Tue, 7 May 2013 19:44:48 +0000
Message-ID: <393252F14C42F946952F1ED75D316CAD385EE2B4@CHEXMBX2A.CHBOSTON.ORG>
References: <51895365.1020707@childrens.harvard.edu>
 <CAOf_dRkkhBT1Fhpc5HsiciCcf_78+B2aP2necW+bdiU7=T1TPQ@mail.gmail.com>
In-Reply-To: 
 <CAOf_dRkkhBT1Fhpc5HsiciCcf_78+B2aP2necW+bdiU7=T1TPQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I don't think that File instantiation is more slow than the ae process, and=
 Tim is talking about tens of thousands of files in the directory tree. =20

The only filesystem call that should exist in any new File(..) is a normali=
ze(..) or resolve(..) on the passed parameter(s), which should just be stri=
ng manipulation and no actual io calls, native or otherwise.  In other word=
s, new File(..) should be fast.

-----Original Message-----
From: ksarma@gmail.com [mailto:ksarma@gmail.com] On Behalf Of Karthik Sarma
Sent: Tuesday, May 07, 2013 3:26 PM
To: dev@ctakes.apache.org
Subject: Re: files vs strings in collection reader

Hmm, without having actually reviewed the code in cTAKES (I'm not on my wor=
k computer), my understanding of the "correct" way of doing this is to use =
the listFiles method on the directory File to get an array of Files; this s=
hould be implemented natively by the JVM and could be faster than individua=
l initialization.


--
Karthik Sarma
UCLA Medical Scientist Training Program Class of 20??
Member, UCLA Medical Imaging & Informatics Lab Member, CA Delegation to the=
 House of Delegates of the American Medical Association ksarma@ksarma.com
gchat: ksarma@gmail.com
linkedin: www.linkedin.com/in/ksarma


On Tue, May 7, 2013 at 12:17 PM, Tim Miller < timothy.miller@childrens.harv=
ard.edu> wrote:

> The FilesInDirectoryCollectionRead**er creates an arraylist of=20
> java.io.File objects when it is initialized. For large datasets (~50k
> files) this is substantial time overhead and probably memory as well.=20
> Seems like it would be more efficient to use Strings instead of Files=20
> there and just open the File object when getNext() is called. It is=20
> pretty easy to implement, any downside to making this switch?
> Tim
>