avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliot Crosby-McCullough <elliot.crosby-mccullo...@freeagent.com>
Subject Re: Parsed JSON schemas not incorporating logical types
Date Thu, 30 Mar 2017 11:18:02 GMT
Hi Pieter,

Thanks for the `getProp` pointer, I've switched our code over.

That ticket is not from me so that's not my schema, however I've reviewed
our schemas and we were doing the same thing.  Perhaps we should update the
docs here to indicate that it needs to be nested within a `type` field:
https://avro.apache.org/docs/1.8.1/spec.html#Logical+Types

I can see that the spec says "Language implementations must ignore unknown
logical types when reading, and should use the underlying Avro type" which
explains your suggestion that `getLogicalType` should the logical type for
built-in types, but it doesn't:

Schema snippet:

{
  "name": "date_field",
  "type": {
    "type": "long",
    "logicalType": "date"
  }
},
{
  "name": "iso8601_date_field",
  "type": {
    "type": "string",
    "logicalType": "iso8601_date"
  }
},
{
  "name": "iso8601_datetime_field",
  "type": {
    "type": "string",
    "logicalType": "iso8601_datetime"
  }
}

Output:

>From getProp: date
>From getLogicalType: null

>From getProp: iso8601_date
>From getLogicalType: null

>From getProp: iso8601_datetime
>From getLogicalType: null

I would be happy to use the built-in logical types but it seems that while
Redshift supports ingesting Avro files, it doesn't seem to know how to
convert Avro dates and times into its own date and time fields.

Maybe that was caused by the above misconfiguration, I'll double check and
provide an update.

On 30 March 2017 at 11:14, Dekinder, Pieter (Nokia - BE/Antwerp) <
pieter.dekinder@nokia.com> wrote:

> Hi Elliot,
>
>
>
> I also came across this inconvenience. I’m using `field.schema.getProp(
> LogicalType.LOGICAL_TYPE_PROP)` for that.
>
> This one is not deprecated.
>
>
>
> Are you using predefined logical types or custom ones?
>
> I would expect the predefined ones to be retrievable with `field.schema().
> getLogicalType().getName()`.
>
>
>
> In the referenced post the JSON was also not really correctly formatted I
> think.
>
> I think It should have been the below… wrapping the logic type related
> things in yet another set of curly braces.
>
>
>
> {
>
>  "namespace": "example.avro",
>
>  "type": "record",
>
>  "name": "User",
>
>  "fields": [
>
>      {
>
>        "name": "name", "type": "string"
>
>      },
>
>      {
>
>        "name": "account_balance",
>
> *       "type": *
>
> *       {*
>
>          "type": "bytes",
>
>          "logicalType": "decimal",
>
>          "precision": 4,
>
>          "scale": 2
>
> *       }*
>
>      }
>
>  ]
>
> }
>
>
>
> Regards,
>
> Pieter
>
>
>
> *From:* Elliot Crosby-McCullough [mailto:elliot.crosby-
> mccullough@freeagent.com]
> *Sent:* woensdag 29 maart 2017 16:37
> *To:* user@avro.apache.org
> *Subject:* Parsed JSON schemas not incorporating logical types
>
>
>
> Hi,
>
>
>
> As noted in this thread from last year[1], Avro seems not to incorporate
> the logical types of fields when parsing JSON files.
>
>
>
> This means that instead of using `field.schema().getLogicalType().getName()`
> I have to use the deprecated `field.props().get("logicalType")` (where
> `field` is a `Schema.Field` retrieved via the record's `schema.getFields()`
> method).
>
>
>
> Any idea why this might be the case?  I'm using Avro 1.8.1 for this.
>
>
>
> Regards,
>
> Elliot
>
>
>
> [1] http://apache-avro.679487.n3.nabble.com/Decimal-types-in-
> IDL-and-generated-Java-not-carrying-logical-types-to-
> schemas-td4034919.html
>

Mime
View raw message