commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Fortner <phidia...@gmail.com>
Subject Re: [CSV] Headers and the first record
Date Wed, 31 Jul 2013 15:14:36 GMT
I took a brief look at the API for CSV, and thought I would share a typical
use case from the biotech industry.  We deal with a lot of instruments that
produce a multiline header.  The header usually contains "experiment
conditions".  You can think of this as metadata for the columnar data.  The
experiment conditions usually contain things like the name of the scientist
using the instrument, the time of day the experiment was run, and some
instrument configuration settings.  Usually when we parse CSV files, we
have to parse the header first, extract all relevant data, and then parse
the rows of data.

In addition to the experiment conditions header, there are also column
headers.  The column headers can be multi-lined as well.  For example, you
might have a column header whose first line contains chemical compound IDs
or names, and the second line of the column header contains the
concentrations for those compounds. The data values represent the percent
inhibition at those concentrations. Like this:

Erlotinib
1uM 10 uM 100 uM 1nM
0.01  0.001  0.0001 0.00001
...

Since the position and types of header and body data vary, we typically use
 parse configuration files that describe "what data can be found where".
 The parse configuration varies not only per instrument but also per
experimental protocol. So there are usually numerous configuration files in
your typical lab.  The configuration files can also be stored in a
database.  This is usually part of a file-watching web app.  It allows
scientists to add support for new experiments or instruments without having
to get a developer to write more code.

In the API I saw support for hard-coded configurations via the CSVFormat
object, but I didn't see any support for creating and using persistable
configurations.  You may want to consider that as you move forward.

Hope this helps,

Mark



On Wed, Jul 31, 2013 at 6:36 AM, Gary Gregory <garydgregory@gmail.com>wrote:

> On Wed, Jul 31, 2013 at 8:58 AM, Gary Gregory <garydgregory@gmail.com
> >wrote:
>
> > On Jul 31, 2013, at 3:38, Benedikt Ritter <britter@apache.org> wrote:
> >
> > > 2013/7/31 Gary Gregory <garydgregory@gmail.com>
> > >
> > >> On Tue, Jul 30, 2013 at 5:29 PM, Emmanuel Bourg <ebourg@apache.org>
> > wrote:
> > >>
> > >>> Le 30/07/2013 23:26, Gary Gregory a écrit :
> > >>>> And another thing: internally, the header should be a Set<String>,
> not
> > >> a
> > >>>> String[]. I plan on fixing that later too.
> > >>>
> > >>> Why should it be a set? Is there an impact on the performance?
> > >>
> > >> Well, I did not finish my though on that one, sorry about that, please
> > >> allow me to walk through my use cases. The issue is about the feature,
> > not
> > >> performance.
> > >>
> > >> At first glance, using a set avoids an inherent problem with any
> non-set
> > >> data structure: defining duplicates. What does the following mean?
> > >>
> > >> withHeader("A", "B", "C", "A");
> > >>
> > >> It's is a recipe for garbage results: record.get("A") returns what?
> > >>
> > >> Today, I added some CSVFormat validation code that checks for
> duplicate
> > >> column names. If you build a format with withHeader("A", "B", "C",
> "A");
> > >> you will get an ISE when validate() is called.
> > >>
> > >> If we had withHeader(Set) and document it as the 'main' way to specify
> > >> column names, then we can say that withHeader(String...) is just a
> > >> syntactical convenience and turn the String[] into a Set. But that
> will
> > not
> > >> work.
> > >>
> > >> The problem with a Java Set is that it is not ordered and the current
> > >> implementation relies on order of the String[]. But why? What the
> > current
> > >> implementation says is: ignore what the header line of the file is and
> > use
> > >> the given column names at the given positions. A perfectly good user
> > story.
> > >> So for withHeader("A", "B", "C"), "A" is column 0, "B" is column 1,
> and
> > so
> > >> on. Ok, that's one usage.
> > >>
> > >> Taking a step back, I want to talk about why should the column name
> > order
> > >> matter when you are calling withHeader(). I would like to be able to
> > tell
> > >> the parser that I want to use a Set of column names and have it figure
> > out,
> > >> based on the header line, the columns indices. This is quite different
> > than
> > >> what we have now.
> > >>
> > >> A use case I have now is a CSV file with a lot of columns (~90) but I
> > only
> > >> care about a small subset of the columns (~10). I'd like to be able to
> > say
> > >> withHeader(Set) where the Set may be a subset of the actual column
> > names in
> > >> the header line. This is different from withHeader(String[]) because
> the
> > >> names in the Set must match the names in the header record.
> > >
> > > I'm not sure if we should try to build in all this different cases
> > > (guessing headers, using the first record as headers, only use a subset
> > of
> > > the available headers) into one implementation.
> > >
> > > What you are talking about sounds more like a view or a projection of
> the
> > > actual content being parsed.
> > > Do we really need this for 1.0 or can it be postponed?
> >
> > This is a real scenario and a real need, not some imaginary complication
> ;)
> >
>
> But I could work with current framework and use withHeaders(new String[]{})
> and let the parser find the headers. Then I can just do record.get("A")
> with the columns I care about. It just feels a little more mysterious.
>
> I think the only wrinkle left for me is that I want validation that the
> columns I care about are there. Right now get(String) throws
> IllegalArgumentException if you give it an unknown column, which will fail
> fast enough on the first record.
>
> So I'll go down that road until the next speed bump...
>
> Gary
>
>
> >
> > Even if it is not implemented for 1.0, we should talk about how it
> > should be done such that it fits in and does not cause API problems
> > later. And if I can get it done by then, then that much the better.
> >
> > Gary
> >
> > >
> > >
> > >>
> > >> So I think it boils down to ignoring my comment about using a Set
> > >> internally and adding a feature where I can tell the parser that I
> want
> > to
> > >> use a set of column names and not worry about the order, because the
> > parser
> > >> will match up the column names when it reads the header line.
> > >>
> > >> Gary
> > >>
> > >>
> > >>>
> > >>>
> > >>> Emmanuel Bourg
> > >>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > >>> For additional commands, e-mail: dev-help@commons.apache.org
> > >>
> > >>
> > >> --
> > >> E-Mail: garydgregory@gmail.com | ggregory@apache.org
> > >> Java Persistence with Hibernate, Second Edition<
> > >> http://www.manning.com/bauer3/>
> > >> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> > >> Spring Batch in Action <http://www.manning.com/templier/>
> > >> Blog: http://garygregory.wordpress.com
> > >> Home: http://garygregory.com/
> > >> Tweet! http://twitter.com/GaryGregory
> > >
> > >
> > >
> > > --
> > > http://people.apache.org/~britter/
> > > http://www.systemoutprintln.de/
> > > http://twitter.com/BenediktRitter
> > > http://github.com/britter
> >
>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org
> Java Persistence with Hibernate, Second Edition<
> http://www.manning.com/bauer3/>
> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> Spring Batch in Action <http://www.manning.com/templier/>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message