drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-951) CSV header row should be parsed
Date Tue, 03 Nov 2015 07:21:27 GMT

    [ https://issues.apache.org/jira/browse/DRILL-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986832#comment-14986832
] 

ASF GitHub Bot commented on DRILL-951:
--------------------------------------

Github user abhipol commented on a diff in the pull request:

    https://github.com/apache/drill/pull/232#discussion_r43721297
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/CompliantTextRecordReader.java
---
    @@ -101,16 +108,23 @@ public void setup(OperatorContext context, OutputMutator outputMutator)
throws E
           InputStream stream = dfs.openPossiblyCompressedStream(split.getPath());
           TextInput input = new TextInput(settings,  stream, readBuffer, split.getStart(),
split.getStart() + split.getLength());
     
    -      TextOutput output = null;
           if(settings.isUseRepeatedVarChar()){
    -        output = new RepeatedVarCharOutput(outputMutator, getColumns(), isStarQuery());
    -      }else{
    -        //TODO: Add field output.
    -        throw new UnsupportedOperationException();
    +        TextOutput output = new RepeatedVarCharOutput(outputMutator, getColumns(), isStarQuery());
    +        this.reader = new TextReader(settings, input, output, whitespaceBuffer);
    +        reader.start();
    +      } else {
    +        // two-phase read approach.
    +        // phase-1: read the header from the file
    +        String [] fieldNames = extractHeader(input);
    +
    +        // phase-2: now read the data
    +        stream = dfs.openPossiblyCompressedStream(split.getPath());
    +        input = new TextInput(settings,  stream, readBuffer, split.getStart(), split.getStart()
+ split.getLength());
    +        TextOutput output = new FieldVarCharOutput(outputMutator, fieldNames, getColumns(),
isStarQuery());
    +        this.reader = new TextReader(settings, input, output, whitespaceBuffer);
    --- End diff --
    
    the second output is using FieldVarCharOutput
    I made few cosmetic changes for better flow. will upload with other changes.


> CSV header row should be parsed
> -------------------------------
>
>                 Key: DRILL-951
>                 URL: https://issues.apache.org/jira/browse/DRILL-951
>             Project: Apache Drill
>          Issue Type: New Feature
>          Components: Storage - Text & CSV
>            Reporter: Tomer Shiran
>            Assignee: Abhijit Pol
>             Fix For: Future
>
>
> CSV reader is currently treating header names like regular rows. There should be a way
to treat the header row as the column names (optional?).
> I exported this dataset to a CSV: https://data.sfgov.org/Public-Safety/SFPD-Incidents-Previous-Three-Months/tmnf-yvry



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message