Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
Received-SPF: pass (athena.apache.org: domain of bruce.mitchener@gmail.com
 designates 209.85.220.49 as permitted sender)
From: Bruce Mitchener <bruce.mitchener@gmail.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-FA4C4794-8C04-43C8-9466-CF9C5BB6C5FD
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)
Subject: Re: Python avro performance
Message-Id: <C359F076-D4D5-4EC8-9FE3-258655CC843C@gmail.com>
Date: Fri, 9 Jan 2015 21:05:54 +0700
References: 
 <CA+ndhHozTsfBzpJrkgGP3dWZHb+cmMP6oqYeG1qjZPhROF1zMA@mail.gmail.com>
 <65ABBE4B-A5E3-4CF4-9E9B-5766D1CFD06B@gmail.com>
 <CA+ndhHqNp03MSbRXkMSr1EmGejqXUt=FDpSTZF9OShRbVRG8Eg@mail.gmail.com>
In-Reply-To: 
 <CA+ndhHqNp03MSbRXkMSr1EmGejqXUt=FDpSTZF9OShRbVRG8Eg@mail.gmail.com>
To: "user@avro.apache.org" <user@avro.apache.org>


--Apple-Mail-FA4C4794-8C04-43C8-9466-CF9C5BB6C5FD
Content-Type: text/plain;
	charset=gb2312
Content-Transfer-Encoding: quoted-printable

Has anyone profiled the Python code or otherwise looked at the performance?

 - Bruce

Sent from my iPhone

> On Jan 9, 2015, at 8:56 PM, Han JU <ju.han.felix@gmail.com> wrote:
>=20
> Hi,=20
>=20
> Thanks. I've tried this project and its performance approaches java/scala.=
 But it seems that it has only read support. We have indeed lots of use case=
s where python program need to persist datasets.=20
>=20
> 2015-01-09 14:39 GMT+01:00 Mika Ristimaki <mika.ristimaki@gmail.com>:
>> Hi,
>>=20
>> I can=A1=AFt really comment why Python Avro is slow but you could try fas=
tavro.
>>=20
>> https://pypi.python.org/pypi/fastavro
>>=20
>> -Mika
>>=20
>>> On 09 Jan 2015, at 15:32, Han JU <ju.han.felix@gmail.com> wrote:
>>>=20
>>> Hi,
>>>=20
>>> I'm evaluating Avro to replace our csv based datasets and I notice a per=
formance problem in avro python bindings.
>>> Basically I've tested on a 1.8GB dataset with 5 columns. With scala (avr=
o java bindings), reads and writes are fast (18s, 44s) but in python, for th=
e same file, it took nearly one hour to write, and 50 miniutes to read ...
>>>=20
>>> My code is based on the avro documentation examples, and the schema is r=
elatively simple. My question:=20
>>>   - Is this performance difference a known issue?=20
>>>   - Is there something I miss (say a special configuration or something)=
?
>>>=20
>>> I've seen a fastavro project and that is much faster in reading, but not=
 write support. This will prevent us from using Avro since we've lot of pyth=
on based programs that need to persist data.
>>>=20
>>> Thanks!
>>> --=20
>>> JU Han
>>>=20
>>> Data Engineer @ Botify.com
>>>=20
>>> +33 0619608888
>=20
>=20
>=20
> --=20
> JU Han
>=20
> Data Engineer @ Botify.com
>=20
> +33 0619608888

--Apple-Mail-FA4C4794-8C04-43C8-9466-CF9C5BB6C5FD
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D=
utf-8"></head><body dir=3D"auto"><div>Has anyone profiled the Python code or=
 otherwise looked at the performance?</div><div><br></div><div>&nbsp;- Bruce=
<br><br>Sent from my iPhone</div><div><br>On Jan 9, 2015, at 8:56 PM, Han JU=
 &lt;<a href=3D"mailto:ju.han.felix@gmail.com">ju.han.felix@gmail.com</a>&gt=
; wrote:<br><br></div><blockquote type=3D"cite"><div><div dir=3D"ltr">Hi,&nb=
sp;<div><br></div><div>Thanks. I've tried this project and its performance a=
pproaches java/scala. But it seems that it has only read support. We have in=
deed lots of use cases where python program need to persist datasets.&nbsp;<=
br><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">2015-01-09 14:3=
9 GMT+01:00 Mika Ristimaki <span dir=3D"ltr">&lt;<a href=3D"mailto:mika.rist=
imaki@gmail.com" target=3D"_blank">mika.ristimaki@gmail.com</a>&gt;</span>:<=
br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-word">Hi,<div=
><br></div><div>I can=E2=80=99t really comment why Python Avro is slow but y=
ou could try fastavro.</div><div><br></div><div><a href=3D"https://pypi.pyth=
on.org/pypi/fastavro" target=3D"_blank">https://pypi.python.org/pypi/fastavr=
o</a></div><span class=3D"HOEnZb"><font color=3D"#888888"><div><br></div><di=
v>-Mika</div></font></span><div><div class=3D"h5"><div><br><div><blockquote t=
ype=3D"cite"><div>On 09 Jan 2015, at 15:32, Han JU &lt;<a href=3D"mailto:ju.=
han.felix@gmail.com" target=3D"_blank">ju.han.felix@gmail.com</a>&gt; wrote:=
</div><br><div><div dir=3D"ltr">Hi,<div><br></div><div>I'm evaluating Avro t=
o replace our csv based datasets and I notice a performance problem in avro p=
ython bindings.</div><div>Basically I've tested on a 1.8GB dataset with 5 co=
lumns. With scala (avro java bindings), reads and writes are fast (18s, 44s)=
 but in python, for the same file, it took nearly one hour to write, and 50 m=
iniutes to read ...</div><div><br></div><div>My code is based on the avro do=
cumentation examples, and the schema is relatively simple. My question:&nbsp=
;</div><div>&nbsp; - Is this performance difference a known issue?&nbsp;</di=
v><div>&nbsp; - Is there something I miss (say a special configuration or so=
mething)?<br clear=3D"all"><div><br></div><div>I've seen a fastavro project a=
nd that is much faster in reading, but not write support. This will prevent u=
s from using Avro since we've lot of python based programs that need to pers=
ist data.</div><div><br></div><div>Thanks!</div>-- <br><div><div dir=3D"ltr"=
><font face=3D"verdana, sans-serif"><b>JU Han</b></font><div><br></div><div>=
<div><div><div>Data Engineer @ <a href=3D"http://Botify.com" target=3D"_blan=
k">Botify.com</a><br></div><div><br></div><div><div><font face=3D"verdana, s=
ans-serif"><a href=3D"tel:%2B33%200619608888" value=3D"+33619608888" target=3D=
"_blank">+33 0619608888</a></font></div></div></div></div></div></div></div>=

</div></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br>=
<br clear=3D"all"><div><br></div>-- <br><div class=3D"gmail_signature"><div d=
ir=3D"ltr"><font face=3D"verdana, sans-serif"><b>JU Han</b></font><div><br><=
/div><div><div><div><div>Data Engineer @ <a href=3D"http://Botify.com">Botif=
y.com</a><br></div><div><br></div><div><div><font face=3D"verdana, sans-seri=
f">+33 0619608888</font></div></div></div></div></div></div></div>
</div></div></div>
</div></blockquote></body></html>=

--Apple-Mail-FA4C4794-8C04-43C8-9466-CF9C5BB6C5FD--