Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6C4DD200CBD for ; Thu, 6 Jul 2017 20:24:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6AE24167174; Thu, 6 Jul 2017 18:24:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 89297167171 for ; Thu, 6 Jul 2017 20:24:05 +0200 (CEST) Received: (qmail 84863 invoked by uid 500); 6 Jul 2017 18:24:04 -0000 Mailing-List: contact dev-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@avro.apache.org Delivered-To: mailing list dev@avro.apache.org Received: (qmail 84846 invoked by uid 99); 6 Jul 2017 18:24:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jul 2017 18:24:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 3B881191738 for ; Thu, 6 Jul 2017 18:24:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id s2XrjoPxpQrz for ; Thu, 6 Jul 2017 18:24:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id E69C95FBBB for ; Thu, 6 Jul 2017 18:24:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 70A28E0732 for ; Thu, 6 Jul 2017 18:24:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2F0B52466E for ; Thu, 6 Jul 2017 18:24:00 +0000 (UTC) Date: Thu, 6 Jul 2017 18:24:00 +0000 (UTC) From: "Manvendra Singh (JIRA)" To: dev@avro.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AVRO-2046) avro-python3: Very restricted set of data types which are allowed in AvroSchemaFromJSONData MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 06 Jul 2017 18:24:06 -0000 [ https://issues.apache.org/jira/browse/AVRO-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manvendra Singh updated AVRO-2046: ---------------------------------- Description: Hey, I come from [CWL project](https://github.com/common-workflow-language/cwltool) and as a part of my GSoC project, I'm working on adding Python 3 compatibility to ``cwltool`` codebase. We've been using avro-python2 for a long time now and it has worked great for us in our projects: schema_salad and cwltool. In the process of porting cwltool, I'm facing issues with avro-python3 library. This is one of the bug I've found in the process. Minimal reproducable example: {code:none} from collections import OrderedDict import avro.schema AvroSchemaFromJSONData = avro.schema.SchemaFromJSONData a = { "fields": [ { "name": "name", "type": "string" }, { "name": "favorite_number", "type": [ "int", "null" ] }, { "name": "favorite_color", "type": [ "string", "null" ] } ], "name": "User", "namespace": "example.avro", "type": "record" } b = OrderedDict(a) AvroSchemaFromJSONData(a) AvroSchemaFromJSONData(b) {code} Ouput: {code} ~/Desktop/test/venv3/lib/python3.5/site-packages/avro/schema.py in SchemaFromJSONData(json_data, names) 1252 if parser is None: 1253 raise SchemaParseException( -> 1254 'Invalid JSON descriptor for an Avro schema: %r.' % json_data) 1255 return parser(json_data, names=names) 1256 SchemaParseException: Invalid JSON descriptor for an Avro schema: OrderedDict([('namespace', 'example.avro'), ('type', 'record'), ('name', 'User'), ('fields', [{'type': 'string', 'name': 'name'}, {'type': ['int', 'null'], 'name': 'favorite_number'}, {'type': ['string', 'null'], 'name': 'favorite_color'}])]). {code} h5. Current implementation of this function does not allow for *any dict like data type*. It however works in avro-python2. Relevant line of code: https://github.com/apache/avro/blob/master/lang/py3/avro/schema.py#L1250 was: Minimal reproducable example: {code:none} from collections import OrderedDict import avro.schema AvroSchemaFromJSONData = avro.schema.SchemaFromJSONData a = { "fields": [ { "name": "name", "type": "string" }, { "name": "favorite_number", "type": [ "int", "null" ] }, { "name": "favorite_color", "type": [ "string", "null" ] } ], "name": "User", "namespace": "example.avro", "type": "record" } b = OrderedDict(a) AvroSchemaFromJSONData(a) AvroSchemaFromJSONData(b) {code} Ouput: {code} ~/Desktop/test/venv3/lib/python3.5/site-packages/avro/schema.py in SchemaFromJSONData(json_data, names) 1252 if parser is None: 1253 raise SchemaParseException( -> 1254 'Invalid JSON descriptor for an Avro schema: %r.' % json_data) 1255 return parser(json_data, names=names) 1256 SchemaParseException: Invalid JSON descriptor for an Avro schema: OrderedDict([('namespace', 'example.avro'), ('type', 'record'), ('name', 'User'), ('fields', [{'type': 'string', 'name': 'name'}, {'type': ['int', 'null'], 'name': 'favorite_number'}, {'type': ['string', 'null'], 'name': 'favorite_color'}])]). {code} h5. Current implementation of this function does not allow for *any dict like data type*. It however works in avro-python2. Relevant line of code: https://github.com/apache/avro/blob/master/lang/py3/avro/schema.py#L1250 > avro-python3: Very restricted set of data types which are allowed in AvroSchemaFromJSONData > ------------------------------------------------------------------------------------------- > > Key: AVRO-2046 > URL: https://issues.apache.org/jira/browse/AVRO-2046 > Project: Avro > Issue Type: Bug > Components: python > Affects Versions: 1.8.2 > Environment: avro-python3 (1.8.2) > Reporter: Manvendra Singh > > Hey, I come from [CWL project](https://github.com/common-workflow-language/cwltool) and as a part of my GSoC project, I'm working on adding Python 3 compatibility to ``cwltool`` codebase. We've been using avro-python2 for a long time now and it has worked great for us in our projects: schema_salad and cwltool. > In the process of porting cwltool, I'm facing issues with avro-python3 library. This is one of the bug I've found in the process. > Minimal reproducable example: > {code:none} > from collections import OrderedDict > import avro.schema > AvroSchemaFromJSONData = avro.schema.SchemaFromJSONData > a = { > "fields": [ > { > "name": "name", > "type": "string" > }, > { > "name": "favorite_number", > "type": [ > "int", > "null" > ] > }, > { > "name": "favorite_color", > "type": [ > "string", > "null" > ] > } > ], > "name": "User", > "namespace": "example.avro", > "type": "record" > } > b = OrderedDict(a) > AvroSchemaFromJSONData(a) > AvroSchemaFromJSONData(b) > {code} > Ouput: > {code} > ~/Desktop/test/venv3/lib/python3.5/site-packages/avro/schema.py in SchemaFromJSONData(json_data, names) > 1252 if parser is None: > 1253 raise SchemaParseException( > -> 1254 'Invalid JSON descriptor for an Avro schema: %r.' % json_data) > 1255 return parser(json_data, names=names) > 1256 > SchemaParseException: Invalid JSON descriptor for an Avro schema: OrderedDict([('namespace', 'example.avro'), ('type', 'record'), ('name', 'User'), ('fields', [{'type': 'string', 'name': 'name'}, {'type': ['int', 'null'], 'name': 'favorite_number'}, {'type': ['string', 'null'], 'name': 'favorite_color'}])]). > {code} > > h5. Current implementation of this function does not allow for *any dict like data type*. It however works in avro-python2. > Relevant line of code: https://github.com/apache/avro/blob/master/lang/py3/avro/schema.py#L1250 -- This message was sent by Atlassian JIRA (v6.4.14#64029)