orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Quanlong Huang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ORC-452) Support converting MAP column from JSON to ORC
Date Tue, 25 Dec 2018 13:07:00 GMT
Quanlong Huang created ORC-452:

             Summary: Support converting MAP column from JSON to ORC
                 Key: ORC-452
                 URL: https://issues.apache.org/jira/browse/ORC-452
             Project: ORC
          Issue Type: New Feature
          Components: Java, tools
            Reporter: Quanlong Huang

The JSON convert tool has not supported MAP column yet.
java -jar orc-tools-1.5.4-uber.jar convert -s "struct<id:int,map_col:map<string,string>>"
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Processing map.json
Exception in thread "main" java.lang.IllegalArgumentException: Unhandled type map<string,string>
        at org.apache.orc.tools.convert.JsonReader.createConverter(JsonReader.java:252)
        at org.apache.orc.tools.convert.JsonReader.<init>(JsonReader.java:277)
        at org.apache.orc.tools.convert.JsonReader.<init>(JsonReader.java:260)
        at org.apache.orc.tools.convert.ConvertTool$FileInformation.getRecordReader(ConvertTool.java:149)
        at org.apache.orc.tools.convert.ConvertTool.run(ConvertTool.java:203)
        at org.apache.orc.tools.convert.ConvertTool.main(ConvertTool.java:165)
        at org.apache.orc.tools.Driver.main(Driver.java:113)
In JsonReader.java, there's no converter for MAP type.

We'd like to convert following JSON rows into an ORC file
{"id": 0, "map_col": {"k1": "v1", "k2": "v2"}}
{"id": 1, "map_col": {"k3": "v3", "k4": "v4", "k5": "v5"}}{code}

This message was sent by Atlassian JIRA

View raw message