hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13970) garbage data read from the beginning of a tar file
Date Fri, 13 Jan 2017 21:54:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822403#comment-15822403
] 

Steve Yang commented on HADOOP-13970:
-------------------------------------

Not really, the contents of the data files are read in correctly. The only garbage text comes
at the beginning of the first line.

> garbage data read from the beginning of a tar file
> --------------------------------------------------
>
>                 Key: HADOOP-13970
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13970
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: common
>    Affects Versions: 2.6.0
>         Environment: Linux EL6
>            Reporter: Steve Yang
>         Attachments: taxi_simplified.tar
>
>
> Hadoop from CDH 5.7.1
> on Spark using databricks ('com.databricks:spark-csv_2.10:1.5.0') to read in a tar file
which consists of 3 .csv files. 
> sqlCtx.read().format("com.databricks.spark.csv").option(...)
> .load(objectName);
> The tar file contains 3 files:
> taxi_simplified1.csv
> taxi2.csv
> simplified3.csv
> where the first line (header) is:
> trip_distance,dropoff_datetime,dropoff_geocode,passenger_count,medallion,rate_code,tip_amount,total_amount,store_and_fwd_flag,mta_tax,pickup_geocode,trip_time_in_secs,surcharge,vendor_id,tolls_amount,fare_amount,pickup_datetime,hack_license,payment_type,ordertime
> Note the first column header is "trip_distance". But the read data shows:
> taxi_simplified1.csv^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@0000644^@0010013^@3001121^@00000046004^@13002371150^@013521^@
0^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ustar
 ^@optitest^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@trip_distance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message