Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0163C200BF4 for ; Fri, 6 Jan 2017 20:48:48 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id F4036160B39; Fri, 6 Jan 2017 19:48:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C8B9F160B37 for ; Fri, 6 Jan 2017 20:48:46 +0100 (CET) Received: (qmail 89949 invoked by uid 500); 6 Jan 2017 19:48:45 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 89939 invoked by uid 99); 6 Jan 2017 19:48:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Jan 2017 19:48:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 169141A0276 for ; Fri, 6 Jan 2017 19:48:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.389 X-Spam-Level: ** X-Spam-Status: No, score=2.389 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=fundingcircle.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id x1hX_q5xu4YB for ; Fri, 6 Jan 2017 19:48:41 +0000 (UTC) Received: from mail-it0-f48.google.com (mail-it0-f48.google.com [209.85.214.48]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 6FD725F1B8 for ; Fri, 6 Jan 2017 19:48:40 +0000 (UTC) Received: by mail-it0-f48.google.com with SMTP id 192so23229877itl.0 for ; Fri, 06 Jan 2017 11:48:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fundingcircle.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=3kUsmnxKyYNq0dB/7gao7N2QiW75vNk4WqNc1vnhGMI=; b=XXBYpID2iDLFBIqtee9+aTZ3dKkGfQNR5sOd0dw7YvNwPb+5w8P7cSTV9HvjFkooT+ tYi2Rcsy27f79epkuNKWImcCSQg1H2lEclzI/0IuZBqknLgyOlu3shY+6pv6FZmiQgBJ ZneaWHrMW152qFNrIStRJ8t19DH844Q0v/xiE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=3kUsmnxKyYNq0dB/7gao7N2QiW75vNk4WqNc1vnhGMI=; b=bzlfMlvSELRkrBGDYB41vtj1be3Ti8iRqWLwIV+TObDqA+FoPtP+xV+qxHik4I8UEw jhpCafTzNNDfiHD4aBb1kQDBY3P1Wv2f8gTBq0rnZKPBSRc5DyePJflYsJsdowS+Zaxm FHDfSiEbug0YO11SRN4Sc4dHzso2rWukCn14SgzPuXouGqRGC6DbqhM2T4ok0jH0md/A 91LvfAbiDRDYI3PjC+R5HTSX/3gIswWJhHvnEPJ7fQ1yDHoCKBJdC3tSP5DPMAZq6l4c dc48K9U6zrjwaEaEMB2glsl3etSDTKG+IS15Dq/BOhjXDDbA5Kxv56I7CGvHCWlmInZy Kovw== X-Gm-Message-State: AIkVDXJkN4A9qVvr4KixBIaXj9lFO15LF0pBJZVHiWhXfve2ghow1MOAaCX1/XX5QbApp+NYb9+d78haDDuY/VDM X-Received: by 10.36.55.202 with SMTP id r193mr242923itr.89.1483732117067; Fri, 06 Jan 2017 11:48:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.16.37 with HTTP; Fri, 6 Jan 2017 11:48:36 -0800 (PST) In-Reply-To: <1530937381.1040397.1479915445025@mail.yahoo.com> References: <1530937381.1040397.1479915445025@mail.yahoo.com> From: Tianxiang Xiong Date: Fri, 6 Jan 2017 11:48:36 -0800 Message-ID: Subject: Re: UUIDs in Avro Schemas To: user@avro.apache.org, Sam Groth Content-Type: multipart/alternative; boundary=001a1140c6f6c818560545724d4d archived-at: Fri, 06 Jan 2017 19:48:48 -0000 --001a1140c6f6c818560545724d4d Content-Type: text/plain; charset=UTF-8 Are the listed logical types those that the Avro parser supports "out of the box", i.e. if using the Avro 1.8.1 Java API, a logical date will be automatically deserialized into a *java.util.Date* and vice-versa? I see that the *LogicalTypes* class has a *uuid()* method, but there is no corresponding *LogicalTypes.UUID* class. There's also no mention of a logical UUID in the documentation . What, then, does the method do? Is there a more detailed example of specifying a custom logical type? I assume that after specifying a logical type in the JSON schema, we can extend the parser so it can do the conversion automatically. There's an example of this for *avsc*, an Avro JavaScript library, but I haven't seen one for Java. On 23 November 2016 at 07:37, Sam Groth wrote: > For UUIDs, I would probably use the fixed type, but a string would also > work. See the specification for the different types that avro supports: > http://avro.apache.org/docs/current/spec.html > > > Sam > > > On Tuesday, November 22, 2016 6:32 PM, Tianxiang Xiong fundingcircle.com> wrote: > > > I see, I'll close the ticket then if using logical types this way to > specify UUID is idiomatic. > > On 22 November 2016 at 16:21, Yibing Shi wrote: > > Hi Tianxiang, > > I am sorry! I replied too quickly! > > I thought that you were complaining that you couldn't generate the UUID > type with IDL compiler, but when I check the JIRA you have just created, I > find that you actually want to know how to specify UUID type in JSON > format, which has already been there. Please use below definition: > > { > "type": "string", > "logicalType": "uuid" > } > > *Yibing Shi* > *Customer Operations Engineer* > > > On Wed, Nov 23, 2016 at 10:43 AM, Tianxiang Xiong fundingcircle.com > wrote: > > Thanks Yibing, I have created AVRO-1962 > . Take a look and let me > know if additional information/changes are needed to the ticket. > > On 22 November 2016 at 15:12, Yibing Shi wrote: > > Hi Tianxiang, > > Looks like that the UUID type hasn't been added to IDL definition yet. > > https://github.com/apache/avro /blob/master/lang/java/compile > r/src/main/javacc/org/apache/a vro/compiler/idl/idl.jj#L214-L 244 > > > Please create a JIRA to track this. > > *Yibing Shi* > *Customer Operations Engineer* > > > On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong .com > wrote: > > The AVRO-1554 > ticket > seems to suggest that the issue of representing UUIDs is resolved with > logical > types > in Avro > 1.8.1. > > The specification offers several examples of using logical types; decimals > are represented as: > > { > > "type": "bytes", > "logicalType": "decimal", > "precision": 4, > "scale": 2 > } > > However, there does not seem to be an example of UUID. The most > popular answers > via > Google/StackOverflow are a few years old and do not make use of > logical types, and can therefore be misleading for newer versions of > Avro > > Is there an idiomatic way of representing UUIDs in Avro Schemas? > > Thanks, > > -- > *Tianxiang Xiong *| Clojure Engineer > *tianxiang.xiong@fundingcircle. com * > 747 Front Street, Floor 4 | San Francisco, CA 94111 > > > > > > -- > *Tianxiang Xiong *| Clojure Engineer > *tianxiang.xiong@fundingcircle. com * > 747 Front Street, Floor 4 | San Francisco, CA 94111 > > > > > > -- > *Tianxiang Xiong *| Clojure Engineer > *tianxiang.xiong@fundingcircle.com * > 747 Front Street, Floor 4 | San Francisco, CA 94111 > > > -- *Tianxiang Xiong *| Clojure Engineer *tianxiang.xiong@fundingcircle.com * 747 Front Street, Floor 4 | San Francisco, CA 94111 --001a1140c6f6c818560545724d4d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Are the listed logical types those that the Avro parser su= pports "out of the box", i.e. if using the Avro 1.8.1 Java API, a= logical date will be automatically deserialized into a java.util.Date=C2=A0and vice-versa?

I see that the LogicalTypes cla= ss has a uuid() method, but there is no corresponding LogicalType= s.UUID class. There's also no mention of a logical UUID in the document= ation. What, then, does the method do?

I= s there a more detailed example of specifying a custom logical type? I assu= me that after specifying a logical type in the JSON schema, we can extend t= he parser so it can do the conversion automatically. There's an example= of this for avsc, an Avro JavaScript library, but I haven't= seen one for Java.

On 23 November 2016 at 07:37, Sam Groth <= ;sgroth@yahoo-inc= .com> wrote:
For UUIDs, I would probably use the = fixed type, but a string would also work. See the specification for the dif= ferent types that avro supports:=C2=A0http://avro.apache.org/docs/current/s= pec.html


Sam


On Tuesday, November 22, 2016 6:32 PM, Tianxian= g Xiong <tianxiang.xiong@fundingcircle.com> wrote:


<= div id=3D"m_-6123948458986410585yiv9233848810">
I see, I'll close the ticket then if using logical types thi= s way to specify UUID is idiomatic.

On 22 November 2016 at 16:21, Yibing Shi <yshi@cloudera.com> wrote:
Hi Tianxiang,

I am sorry! I replied too= quickly!

I thought that you were complainin= g that you couldn't generate the UUID type with IDL compiler, but when = I check the JIRA you have just created, I find that you actually want to kn= ow how to specify UUID type in JSON format, which has already been there. P= lease use below definition:

{
=C2=A0 &= quot;type": "string",
=C2=A0 "logical= Type": "uuid"
}
<= div class=3D"m_-6123948458986410585yiv9233848810gmail_extra">
Yibing Shi
Customer Operations Engineer


On Wed, Nov 23, 2016 at 10:43 AM, Tianxiang Xiong <tianxiang.xiong@ fundingcircle.com> wrote:
Thank= s Yibing, I have created AVRO-1962. = Take a look and let me know if additional information/changes are needed to= the ticket.

On 22 November 2016 at 15:12, Yibing= Shi <yshi@cloudera.com> wro= te:
Hi Tianxiang,
<= br clear=3D"none">
Looks like th= at the UUID type hasn't been added to IDL definition yet.

https://github.com/apache/= avro /blob/master/lang/java/compile r/src/main/javacc/org/apache/a vro/comp= iler/idl/idl.jj#L214-L 244

Please create a JIRA to track this.
=
Yibing S= hi
= Customer Operations Engineer


On Wed, Nov 23, 2016 at 4:58 AM, Tianxiang Xiong <tianxiang.xiong@fundingcircl= e .com> wrote:
The AVRO-1554 <https://issues.apache.org/jir a/browse/AV= RO-1554> ticket
s= eems to suggest that the issue of representing UUIDs is resolved with logic= al
types &= lt;http://avro.apache.org/docs/1 .8.1/spec.html#Logical+Types> in Avro
1.8.1.

The specification offers several examples of using= logical types; decimals
are represented as:

{

=C2=A0 "type": "bytes",
=C2=A0 "logicalType": "decimal",
=C2=A0 "precision": 4,
=C2=A0 "s= cale": 2
}

However, there does not seem to be an example of UUID. The most
popular answers &l= t;http://st= ackoverflow.com/a/19 705149/864684>= via
<= span class=3D"m_-6123948458986410585yiv9233848810m_-1207318562311376517m_-4= 728576757692052864m_-61145753679369976m_3818043305619513536gmail-im" style= =3D"font-size:12.8px">Google/StackOverflow are a few years old and do not m= ake use of
logical types, and can therefore be misleading= for newer versions of
Avro

Is there an idiomatic way of representing UUIDs in Avro Schemas?<= br clear=3D"none">
Thanks,

--
<= div>
Tianxiang Xiong=C2=A0| Clojure Engineer=C2=A0
747 Front Street, Floor 4 | San Francisco, C= A 94111




--
Tianxiang Xiong=C2=A0| Clojure Engineer=C2=A0=
747 Front Street, Floor 4 | San Franci= sco, CA 94111




--


<= /blockquote>


--
--001a1140c6f6c818560545724d4d--