Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Sat, 12 Sep 2015 01:48:45 +0000 (UTC)
From: "Aman Sinha (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.12863262.1441933178000.320770.1442022525660@Atlassian.JIRA>
In-Reply-To: <JIRA.12863262.1441933178000@Atlassian.JIRA>
References: <JIRA.12863262.1441933178000@Atlassian.JIRA>
 <JIRA.12863262.1441933178607@arcas>
Subject: [jira] [Comment Edited] (DRILL-3764) Support the ability to
 identify and/or skip records when a function evaluation fails
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/DRILL-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741847#comment-14741847 ] 

Aman Sinha edited comment on DRILL-3764 at 9/12/15 1:48 AM:
------------------------------------------------------------

There's actually a 4th option : we were discussing with [~vkorukanti] and [~jni]:
 4) If  a function throws an error (say NumberFormatException) then assign the output of that function as null conditional on a system option.  This is similar to what Hive does. 


was (Author: amansinha100):
There's actually a 4th option : we were discussing with [~vkorukanti] and [~jni]:
 4) If  a function evaluation fails then assign the output of that function as null.   This is similar to what Hive does. 

> Support the ability to identify and/or skip records when a function evaluation fails
> ------------------------------------------------------------------------------------
>
>                 Key: DRILL-3764
>                 URL: https://issues.apache.org/jira/browse/DRILL-3764
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Functions - Drill
>    Affects Versions: 1.1.0
>            Reporter: Aman Sinha
>            Assignee: Mehant Baid
>
> Drill can point out the filename and location of corrupted records in a file but it does not have a good mechanism to deal with the following scenario: 
> Consider a text file with 2 records:
> {code}
> $ cat t4.csv
> 10,2001
> 11,http://www.cnn.com
> {code}
> {code}
> 0: jdbc:drill:zk=local> alter session set `exec.errors.verbose` = true;
> 0: jdbc:drill:zk=local> select cast(columns[0] as init), cast(columns[1] as bigint) from dfs.`t4.csv`;
> Error: SYSTEM ERROR: NumberFormatException: http://www.cnn.com
> Fragment 0:0
> [Error Id: 72aad22c-a345-4100-9a57-dcd8436105f7 on 10.250.56.140:31010]
>   (java.lang.NumberFormatException) http://www.cnn.com
>     org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeL():91
>     org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varCharToLong():62
>     org.apache.drill.exec.test.generated.ProjectorGen1.doEval():62
>     org.apache.drill.exec.test.generated.ProjectorGen1.projectRecords():62
>     org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172
> {code}
> The problem is user does not have the context of where the error occurred -either the file name or the record number.   This becomes a pain point especially when CTAS is being used to do data conversion from (say) text format to Parquet format.  The CTAS may be accessing thousands of files and 1 such casting (or another function) failure aborts the query. 
> It would substantially improve the user experience if we provided: 
> 1) the filename and record number where  this failure occurred
> 2) the ability to skip such records depending on a session option
> 3) the ability to write such records to a staging table for future ingestion
> Please see discussion on dev list: 
> http://mail-archives.apache.org/mod_mbox/drill-dev/201509.mbox/%3cCAFyDVvLuPLgTNZ56S6=J=9Vb=aBs=pDw7NRHKkdUPbdxGFAdcg@mail.gmail.com%3e


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)