arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Дима Фомин <dat...@yandex.ru>
Subject Re: ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
Date Fri, 25 Oct 2019 13:46:20 GMT
<div>Hello, Micah!</div><div> </div><div>Gotcha! Seems the problem
in in pyarrow.array() handling of decimal values.</div><div> </div><div><div
style="background:#1e1e1e;color:#d4d4d4;font:14px / 19px 'monospace' , 'consolas' , 'courier
new' , monospace;white-space:pre"><div><span style="color:#d4d4d4">data1 = </span><span
style="color:#dcdcdc">[</span><span style="color:#d4d4d4">decimal.Decimal</span><span
style="color:#dcdcdc">(</span><span style="color:#ce9178">'234234234234.43'</span><span
style="color:#dcdcdc">)]</span></div> <div><span style="color:#6aa94f">#pa.array somehow broke precision and scale?</span></div><div><span
style="color:#d4d4d4">data2 = pa.array</span><span style="color:#dcdcdc">(</span><span
style="color:#d4d4d4">data1</span><span style="color:#dcdcdc">)</span></div> <div><span
style="color:#d4d4d4">sss1 = pa.schema</span><span style="color:#dcdcdc">({<!--
--></span><span style="color:#ce9178">'decimal_col'</span><span style="color:#dcdcdc">:</span><span
style="color:#d4d4d4"> pa.decimal128</span><span style="color:#dcdcdc">(</span><span
style="color:#b5cea8">22</span><span style="color:#dcdcdc">,</span><span
style="color:#b5cea8">2</span><span style="color:#dcdcdc">)})</span></div> <div><span
style="color:#d4d4d4">pa.Table.from_arrays</span><span style="color:#dcdcdc">([</span><span
style="color:#d4d4d4">data1</span><span style="color:#dcdcdc">],</span><span
style="color:#d4d4d4"> schema=sss1</span><span style="color:#dcdcdc">)</span></div></div></div><div> </div><div>is
OK</div><div> </div><div>bur this one</div><div><div
style="background:#1e1e1e;color:#d4d4d4;font:14px / 19px 'monospace' , 'consolas' , 'courier
new' , monospace;white-space:pre"><div><span style="color:#d4d4d4">pa.Table.from_arrays</span><span
style="color:#dcdcdc">([</span><span style="color:#d4d4d4">data2</span><span
style="color:#dcdcdc">],</span><span style="color:#d4d4d4"> schema=sss1</span><span
style="color:#dcdcdc">)</span></div></div></div><div> </div><div>raise
<span style="background:#383838;font:400 14px monospace;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:pre;word-spacing:0px">ArrowNotImplementedError</span><span
style="background:#383838;color:#d5d5d5;font:400 14px monospace;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:pre;word-spacing:0px">:
No cast implemented from decimal(14, 2) to decimal(22, 2)</span></div><div> </div><div>see
it here</div><div>https://colab.research.google.com/drive/1ejOyFBVs8GPFTosuRMDyTwPvt4145rgV</div><div> </div><div>Should
I post in to JIRA as well? Or this is the same reason posted above?</div><div> </div><div> </div><div>25.10.2019,
08:00, "Micah Kornfield" &lt;emkornfield@gmail.com&gt;:</div><blockquote><div>Hi
Dima,<div>Thanks for finding this bug would you mind filing a JIRA (<a href="https://issues.apache.org/jira/secure/Dashboard.jspa">https://issues.apache.org/jira</a>).  </div><div> </div><div>Note
that this code works fine if  you put quotes around the decimal value (e.g. <span style="color:#000000;font:11px
'menlo'">Decimal('123.234')), but as you state it crashes python because the Decimal created
without quotes requires more then 38 decimal places to represent.</span></div><div> </div><div><span
style="color:#000000;font:11px 'menlo'">Thanks,</span></div><div><span
style="color:#000000;font:11px 'menlo'">Micah</span></div><div> </div><div> </div></div> <div><div>On
Thu, Oct 24, 2019 at 7:53 AM Дима Фомин &lt;<a href="mailto:datagy@yandex.ru">datagy@yandex.ru</a>&gt;
wrote:</div><blockquote style="border-left-color:rgb( 204 , 204 , 204 );border-left-style:solid;border-left-width:1px;margin:0px
0px 0px 0.8ex;padding-left:1ex"><div>Digging this I just found that this code breaks
my Python Kernel :(</div><div> </div><div>import sqlalchemy as sq</div><div>import
pyarrow as pa</div><div> </div><div>aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234),
sq.sql.sqltypes.decimal.Decimal(123.234)</div><div>ddd = [aaa, bbb]</div><div>pa.array(ddd)</div><div> </div><div>So
seems pyarrow.array is limited with types of array elements, doesnt'it?</div><div> </div><div>WBR,</div><div>Dima
Fomin</div><div> </div><div>24.10.2019, 15:47, "Dima Fomin" &lt;<a
href="mailto:datagy@yandex.ru">datagy@yandex.ru</a>&gt;:</div><blockquote><div>Hello!</div><div>First
of all thank you very much for your efforts on Apache Arrow! This looks very useful and prominent
tech to be able use it in self-BI area.</div><div> </div><div>I faced
such issues - get Subj Error</div><blockquote><div>ArrowNotImplementedError:
No cast implemented from decimal(2,2) to decimal(12,2)</div></blockquote><div>when
trying to <em>pyarrow.Table.from_batches</em> array of values defined in schema
as <em>pyarrow.decimal128(12,2)</em>.</div><div> </div><div>Which
values are in fact column of <em>sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"</em>
values, which I map in schema like this <em>pyarrow.decimal128(12,2)</em>.</div><div> </div><div>What
could be a reason of issue?</div><div>Could it be that somehow declared decimal
type of data is reduced to minimum value which met in data? :/</div><div> </div><div>With
best regards,</div><div>Dima Fomin</div></blockquote></blockquote></div></blockquote>
Mime
View raw message