avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong, Shunhui Welly" <Shunhui.Welly.H...@jpmorgan.com>
Subject RE: Avro decode very slow in Python
Date Fri, 25 Mar 2016 13:02:38 GMT
Hi Evan,

Thank you for the information. However, it is difficult to find document on pyavroc. I can't
find how pyavroc to decode binary (It took 5mins to decode 55M data in Avro Python). Can you
give me a sample?

Thank you,
Welly

-----Original Message-----
From: Evan McClain [mailto:aeroevan@gmail.com] 
Sent: Thursday, March 24, 2016 9:21 PM
To: user@avro.apache.org
Subject: Re: Avro decode very slow in Python

On Thu, 2016-03-24 at 21:03 +0000, Hong, Shunhui Welly wrote:
> Hi guys,
>  
> I am trying to decode avro binary data (55M took 5 mins) in Python.
> It seems 10+ times slower than in Java. Any suggestions?

I have used both fastavro and pyavroc. pyavroc is a little more trouble to install but should
be fairly close to the java implementation.

fastavro could definitely be improved since it's not even using cython's types. I've been
meaning to look into adding types to fastavro but free time is harder to find these days...

--
Evan McClain <aeroevan@gmail.com>

This email is confidential and subject to important disclaimers and conditions including on
offers for the purchase or sale of securities, accuracy and completeness of information, 
viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email
Mime
View raw message