crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barry,Nathan" <NBA...@CERNER.COM>
Subject Re: Enhancement to CSV input format?
Date Thu, 07 May 2015 16:17:19 GMT
We’ll get a Crunch JIRA logged soon, but at the moment we are debating how
the CSV Input should attempt to recover from the unescaped quote, should
the input

1) look for a proper end quote (where the next char is a delimiter, EOL or
EOF) as recover at that point, mid-line; else fall back to the next EOL
2) look for the next EOL and recover at that point

Example scenario, invalid input text:

this,is,a,"line with 1" problems","in
this,is,another,line of,text

Output using 1 (With ---- as the record breaks)

this,is,a,"line with 1" problems","in
this,is,another,line of,text

Output using 2

this,is,a,"line with 1" problems",”in

this,is,another,line of,text

On 5/6/15, 9:12 AM, "Josh Wills" <> wrote:

>On Wed, May 6, 2015 at 3:04 PM, Barry,Nathan <> wrote:
>> Correct me if I am wrong, but with the current code when an unescaped
>> quote is encountered the code doesn¹t always blow up, rather it become
>> of sequence with the open/close quotes which will often lead to
>> where both the field delimiters and eol markers are now treated as if
>> are inside a quoted attribute (i.e. ignored) resulting in a CSV record
>> that is potentially huge, with the size being determined when the code
>> finds the next unescaped quote, at which point the code will honor the
>> next field delimited/eol marker.
>That seems like the sort of thing worth fixing in the core, IMHO.
>> So in our bad file examples CSV records were created with individual
>> record sizes in the 10s of MB rather than the expected ~3-4KB.
>> What we were looking to do was to:
>> - when currently in a quoted field
>> - if we find another quote
>> - look to the next character and see:
>> - if it¹s a delimiter, eol or eof marker - then close the quote and keep
>> processing normally
>> - if not a delimited, eol or eof marker - then we have a bad record, so
>> ignore all quotes & delimiters and simply look for the next eol/eof and
>> break the record there
>> The thought being that the 1 bad record won¹t corrupt the entire file or
>> corrupt the record splits; though the consumers of each record would
>> encounter errors when trying to parse the record, but they can then
>> determine the best course of action:  ignore it, reject it, reject the
>> whole file, etc.
>> Nathan
>> On 5/5/15, 9:30 AM, "Champion,Mac" <> wrote:
>> >Some users of the CSV Input Format at Cerner had some issues with CSV
>> >files from clients where there were stray, unescaped double-quotes
>> >of fields (ostensibly representing inches). Some bureaucratic stuff
>> >prevented us from getting those files reliably cleaned up, so we
>> >brainstormed and figured out a way to make the CSV Input Format able to
>> >ignore the stray quotes and pass them forward to be handled by whatever
>> >parsing solution comes later. We are working on implementing this into
>> >our copy of the input format and it seems to be working so far.
>> >
>> >My question is, is this something that we should log a JIRA for and
>> >submit our work to Crunch as well? It¹s handy in our case, but the
>> >are truly malformed and not following the CSV standards. Should the
>> >CSVInputFormat have configurable options to be able to handle malformed
>> >files and pass bad records forward, or is the current behavior (blow up
>> >and give some info about where the bad records start) the way it truly
>> >should behave?
>> >
>> >Thanks for your input,
>> >Mac
>> >
>> >CONFIDENTIALITY NOTICE This message and any included attachments are
>> >Cerner Corporation and are intended only for the addressee. The
>> >information contained in this message is confidential and may
>> >inside or non-public information under international, federal, or state
>> >securities laws. Unauthorized forwarding, printing, copying,
>> >distribution, or use of such information is strictly prohibited and may
>> >be unlawful. If you are not the addressee, please promptly delete this
>> >message and notify the sender of the delivery error by e-mail or you
>> >call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1)
>> >(816)221-1024.
>Director of Data Science
>Cloudera <>
>Twitter: @josh_wills <>

View raw message