pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ratandeep Ratti (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4448) AvroMapWrapper leaks Avro data types when the map values are complex avro records
Date Fri, 06 Mar 2015 16:30:38 GMT

    [ https://issues.apache.org/jira/browse/PIG-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350543#comment-14350543
] 

Ratandeep Ratti commented on PIG-4448:
--------------------------------------

For instance, below is the Avro schema and Pig schema for an Avro record which contains a
map which contain a record.
{code}
Avro schema
{
  "type": "record",
  "name": "recordInMap",
  "namespace": "org.apache.pig.test.builtin",
  "fields": [
    {
      "name": "key",
      "type": "string"
    },
    {
      "name": "value",
      "type": "int"
    },
    {
      "name": "parameters",
      "type": {
        "type": "map",
        "values": {
          "type": "record",
          "name": "A",
          "fields": [
            {
              "name": "id",
              "type": [
                "null",
                "int"
              ]
            }
          ]
        }
      }
    }
  ]
}

Pig schema
key:chararray,value:int,parameters:[A:(id:int)]
{code}

According to this, the Pig representation should contain a tuple for the the map's value record,
but instead it currently contains an Avro GenericRecord

Updating the patch. Now I'm using the logic from org.apache.pig.impl.util.avro.AvroBagWrapper
on how it is converting each element of the Bag to an equivalent Pig type.

My only gripe now is that the name of the method which helps to convert types from avro to
Pig in AvroTupleWrapper is "resolveUnion" . Ideally I'd love to name it avroToPig or something
similar but since its scope is  public  I'm refraining to do so.

> AvroMapWrapper leaks Avro data types when the map values are complex avro records
> ---------------------------------------------------------------------------------
>
>                 Key: PIG-4448
>                 URL: https://issues.apache.org/jira/browse/PIG-4448
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.13.0
>            Reporter: Ratandeep Ratti
>            Assignee: Ratandeep Ratti
>             Fix For: 0.15.0
>
>         Attachments: avro_map_wrapper_fix.patch
>
>
> AvroMapWrapper does not convert value objects in it's inner map to Pig types when queried.
This could lead to problems when the value object is a complex Avro type. 
> Attaching a preliminary patch. Please let me know if this is not the best way to fix
it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message