avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dekinder, Pieter (Nokia - BE/Antwerp)" <pieter.dekin...@nokia.com>
Subject RE: Parsed JSON schemas not incorporating logical types
Date Thu, 30 Mar 2017 12:13:21 GMT
Hi Eliot,

I don’t see these logical types in the documentation. So I guess they are not predefined.
(maybe Redshift implements them?)
Indeed the official documentation should be a bit better, but I think there is already a Jira
for that.
Can you check with one of the documented logical types and see if that does work?

Regards,
Pieter

From: Elliot Crosby-McCullough [mailto:elliot.crosby-mccullough@freeagent.com]
Sent: donderdag 30 maart 2017 13:18
To: user@avro.apache.org
Subject: Re: Parsed JSON schemas not incorporating logical types

Hi Pieter,

Thanks for the `getProp` pointer, I've switched our code over.

That ticket is not from me so that's not my schema, however I've reviewed our schemas and
we were doing the same thing.  Perhaps we should update the docs here to indicate that it
needs to be nested within a `type` field: https://avro.apache.org/docs/1.8.1/spec.html#Logical+Types

I can see that the spec says "Language implementations must ignore unknown logical types when
reading, and should use the underlying Avro type" which explains your suggestion that `getLogicalType`
should the logical type for built-in types, but it doesn't:

Schema snippet:

{
  "name": "date_field",
  "type": {
    "type": "long",
    "logicalType": "date"
  }
},
{
  "name": "iso8601_date_field",
  "type": {
    "type": "string",
    "logicalType": "iso8601_date"
  }
},
{
  "name": "iso8601_datetime_field",
  "type": {
    "type": "string",
    "logicalType": "iso8601_datetime"
  }
}
Output:

From getProp: date
From getLogicalType: null

From getProp: iso8601_date
From getLogicalType: null

From getProp: iso8601_datetime
From getLogicalType: null

I would be happy to use the built-in logical types but it seems that while Redshift supports
ingesting Avro files, it doesn't seem to know how to convert Avro dates and times into its
own date and time fields.

Maybe that was caused by the above misconfiguration, I'll double check and provide an update.

On 30 March 2017 at 11:14, Dekinder, Pieter (Nokia - BE/Antwerp) <pieter.dekinder@nokia.com<mailto:pieter.dekinder@nokia.com>>
wrote:
Hi Elliot,

I also came across this inconvenience. I’m using `field.schema.getProp(LogicalType.LOGICAL_TYPE_PROP)`
for that.
This one is not deprecated.

Are you using predefined logical types or custom ones?
I would expect the predefined ones to be retrievable with `field.schema().getLogicalType().getName()`.

In the referenced post the JSON was also not really correctly formatted I think.
I think It should have been the below… wrapping the logic type related things in yet another
set of curly braces.

{
 "namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {
       "name": "name", "type": "string"
     },
     {
       "name": "account_balance",
       "type":
       {
         "type": "bytes",
         "logicalType": "decimal",
         "precision": 4,
         "scale": 2
       }
     }
 ]
}

Regards,
Pieter

From: Elliot Crosby-McCullough [mailto:elliot.crosby-mccullough@freeagent.com<mailto:elliot.crosby-mccullough@freeagent.com>]
Sent: woensdag 29 maart 2017 16:37
To: user@avro.apache.org<mailto:user@avro.apache.org>
Subject: Parsed JSON schemas not incorporating logical types

Hi,

As noted in this thread from last year[1], Avro seems not to incorporate the logical types
of fields when parsing JSON files.

This means that instead of using `field.schema().getLogicalType().getName()` I have to use
the deprecated `field.props().get("logicalType")` (where `field` is a `Schema.Field` retrieved
via the record's `schema.getFields()` method).

Any idea why this might be the case?  I'm using Avro 1.8.1 for this.

Regards,
Elliot

[1] http://apache-avro.679487.n3.nabble.com/Decimal-types-in-IDL-and-generated-Java-not-carrying-logical-types-to-schemas-td4034919.html

Mime
View raw message