Mailing-List: contact dev-help@flink.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of mneumann@spotify.com designates
 209.85.212.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANC1h_uWG-eYku4mo0m5+mmG4AkehkWiM_ScLf_z4OZT56HLSw@mail.gmail.com>
References: 
 <CAHuUK3UMDCdBw8TG+1dJKiDkSW2gMW1a-G0O0PjsmAm4hsjkcQ@mail.gmail.com>
	<CANC1h_uWG-eYku4mo0m5+mmG4AkehkWiM_ScLf_z4OZT56HLSw@mail.gmail.com>
Date: Wed, 15 Oct 2014 16:07:17 +0200
Message-ID: 
 <CAHuUK3WS9Ltb-r-UnJbtfpuZ_T-b7fq38xDGXQb+ALsOdAUetA@mail.gmail.com>
Subject: Re: CsvInputFormat delimiter fields
From: Martin Neumann <mneumann@spotify.com>
To: dev@flink.incubator.apache.org
Content-Type: multipart/alternative; boundary=f46d043d673f45e52e050576a6a7

--f46d043d673f45e52e050576a6a7
Content-Type: text/plain; charset=UTF-8

Would changing it cost performance?
If not I thing it would be a good change to make since it allows to (ab)use
the csv reader to load structured Text files (for example by putting
Keywords as delimiter).

Being able to put a regular expression there would be even nicer but maybe
it should end up in its own InputFormat then.

cheers Martin

On Wed, Oct 15, 2014 at 3:47 PM, Stephan Ewen <sewen@apache.org> wrote:

> Hi!
>
> The reason is the current way the csv parsers work. They are pushed into
> the byte stream parsing and are restricted to recognize one char
> delimiters. It is possible to change that, but would be a bit of work.
>
> Stephan
>
> On Wed, Oct 15, 2014 at 3:36 PM, Martin Neumann <mneumann@spotify.com>
> wrote:
>
> > Hej,
> >
> > A lot of my inputs are csv files so I use the CsvInputFormat a lot. What
> I
> > find kind of odd that the Line delimiter is a String but the Field
> > delimiter is a Character.
> >
> > *see:* new CsvInputFormat<Tuple2<String,String>>(new
> > Path(pVecPath),"\n",'\t',String.class,String.class)
> >
> > Is there a reason for this? I'm currently working with a file that has a
> > more complex field delimiter so I had to write a mapper to read from
> > StringInputFormat.
> >
> > cheers Martin
> >
>

--f46d043d673f45e52e050576a6a7--