spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] cloud-fan commented on issue #26831: [SPARK-30201][SQL] HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT
Date Wed, 11 Dec 2019 14:52:46 GMT
cloud-fan commented on issue #26831: [SPARK-30201][SQL] HiveOutputWriter standardOI should
use ObjectInspectorCopyOption.DEFAULT
URL: https://github.com/apache/spark/pull/26831#issuecomment-564580922
 
 
   > how do you get non-UTF-8 data here in the first place?
   
   This is a good question. once data is load into Spark, string type must be a UTF-8 encoded
binary. If it's not, then data loading (table reading) has a bug.
   
   It's a little weird to hear that: hive table reading is fine, but the writing after reading
is problematic because string is not UTF8 encoded. Seems like the table reading is wrong at
the first place (produce non-utf8-encoded string).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message