Mailing-List: contact commons-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Jakarta Commons Developers List" <commons-dev@jakarta.apache.org>
Received-SPF: pass (asf.osuosl.org: domain of flamefew@gmail.com designates
 64.233.162.199 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:cc:mime-version:content-type:content-transfer-encoding:content-disposition;
        b=q1IL+mbE/E0yWcRy09hP8WMeKOtN+4q+OV5aIJMcXeqPK2XIASSrk+VKUyxrLknC+5atGjNc3thv0KXTNatHLvD/ZNOAv3vAYNDWm3ngV0NGX4Z9sEBwu2Qfg1uV6Mp0chkM5K0T4lYZwLh62Fxlp6bIknBMM0tX4Xlvzkc1F0A=
Message-ID: <31cc37360601142131h6e6b0afavd37743eb5d4ae926@mail.gmail.com>
Date: Sun, 15 Jan 2006 00:31:01 -0500
From: Henri Yandell <flamefew@gmail.com>
To: Jakarta Commons Developers List <commons-dev@jakarta.apache.org>
Subject: [csv] feature and performance analysis
Cc: "Sean C. Sullivan" <sean@seansullivan.com>,
	Steven Caswell <steven.caswell@gmail.com>,
	Brian McCallister <brianm@apache.org>,
	Glen Smith <glen@bytecode.com.au>,
	Stefan Rufer <stefan.rufer@netcetera.ch>,
	Urs Hardegger <urs.hardegger@netcetera.ch>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Spent a little time over the last week doing both performance and
feature set analysis of the 5 open-source CSV libraries that I'm aware
of.

First up, feature sets:

http://people.apache.org/~bayard/commons-csv/csv-features.xhtml

Secondly, performance. The code/data is sitting in:

http://people.apache.org/~bayard/commons-csv/csv-perf/

Run under 1.5 because the Ostermiller library is compiled for 1.5 [grumble]=
.

Results

http://people.apache.org/~bayard/commons-csv/csv-perf/results.csv

Take with a grain of salt, these aren't quite the pattern I was seeing
when running a few days ago on the plane. Ideally needs to run
multiple time and take median or some such.

Generally, Ostermiller is fastest for parsing, with Skife then Open a
chunk behind. GJ a little behind them and Commons lagging by a lot.

On printing, Open edges Skife, GJ a bit behind, Commons and then
Ostermiller lagging a lot.

-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-

What did I learn from this.

* Lots of features out there, no library contains all of them. I don't
think that many of them are mutually exclusive.

* Ostermiller's parser is very quick. Possibly because he's built on
top of JFlex? The printer is very, very slow.

* The current Commons parser is very slow. As is the printer.

* I also did some bug checking. Given that I only had 7 quick lines of
pain, I don't think any parser managed to parse them with much
success: http://people.apache.org/~bayard/commons-csv/csv-perf/dependabilit=
y.csv

Mostly; that despite the odd looks I get when I mention having a
commons-csv (people think they're dumb simple things), we all have
lots of room for improvement.

-----

So what next? Are the poor performance stats for Commons-CSV a worry?
Are they offset enough by having more features? Should we look into a
lexical tool approach as it seems to work for Ostermiller?

Class-wise, I'd like to see something like:

Csv         (instead of using String[][] or List of String[])
CsvPrinter
CsvParser
CsvException
CsvStrategy  (used by both printer and parser)

I don't see any reason to not want every feature in the feature file.

That's it for the night. If you're not on commons-dev, mail
commons-dev-subscribe@jakarta.apache.org to join in. Cc's are unlikely
to last too long on a thread. Make sure you keep the [csv] on the
emails and future ones, useful way of separating the components out.

Hen

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org