arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
Date Fri, 25 Oct 2019 14:44:42 GMT
Apologies, I checked colab and it's 0.15.0 there, I confirm that

In [8]: pa.table([data2], schema=sss1)

does not work due to the missing cast implementation. It would be
great if someone would contribute an implementation for ARROW-6990

On Fri, Oct 25, 2019 at 9:42 AM Wes McKinney <wesmckinn@gmail.com> wrote:
>
> These code examples both work for me in 0.15.0 and master. What
> version of the project are you using?
>
> In [6]: pa.array(data1).type
> Out[6]: Decimal128Type(decimal(14, 2))
>
> In [7]: pa.table([data1], schema=sss1)
> Out[7]:
> pyarrow.Table
> decimal_col: decimal(22, 2)
>
> On Fri, Oct 25, 2019 at 8:46 AM Дима Фомин <datagy@yandex.ru> wrote:
> >
> > Hello, Micah!
> >
> > Gotcha! Seems the problem in in pyarrow.array() handling of decimal values.
> >
> > data1 = [decimal.Decimal('234234234234.43')]
> >
> > #pa.array somehow broke precision and scale?
> > data2 = pa.array(data1)
> >
> > sss1 = pa.schema({'decimal_col': pa.decimal128(22,2)})
> >
> > pa.Table.from_arrays([data1], schema=sss1)
> >
> > is OK
> >
> > bur this one
> > pa.Table.from_arrays([data2], schema=sss1)
> >
> > raise ArrowNotImplementedError: No cast implemented from decimal(14, 2) to decimal(22,
2)
> >
> > see it here
> > https://colab.research.google.com/drive/1ejOyFBVs8GPFTosuRMDyTwPvt4145rgV
> >
> > Should I post in to JIRA as well? Or this is the same reason posted above?
> >
> >
> > 25.10.2019, 08:00, "Micah Kornfield" <emkornfield@gmail.com>:
> >
> > Hi Dima,
> > Thanks for finding this bug would you mind filing a JIRA (https://issues.apache.org/jira).
> >
> > Note that this code works fine if  you put quotes around the decimal value (e.g.
Decimal('123.234')), but as you state it crashes python because the Decimal created without
quotes requires more then 38 decimal places to represent.
> >
> > Thanks,
> > Micah
> >
> >
> >
> > On Thu, Oct 24, 2019 at 7:53 AM Дима Фомин <datagy@yandex.ru> wrote:
> >
> > Digging this I just found that this code breaks my Python Kernel :(
> >
> > import sqlalchemy as sq
> > import pyarrow as pa
> >
> > aaa, bbb = sq.sql.sqltypes.decimal.Decimal(123.234), sq.sql.sqltypes.decimal.Decimal(123.234)
> > ddd = [aaa, bbb]
> > pa.array(ddd)
> >
> > So seems pyarrow.array is limited with types of array elements, doesnt'it?
> >
> > WBR,
> > Dima Fomin
> >
> > 24.10.2019, 15:47, "Dima Fomin" <datagy@yandex.ru>:
> >
> > Hello!
> > First of all thank you very much for your efforts on Apache Arrow! This looks very
useful and prominent tech to be able use it in self-BI area.
> >
> > I faced such issues - get Subj Error
> >
> > ArrowNotImplementedError: No cast implemented from decimal(2,2) to decimal(12,2)
> >
> > when trying to pyarrow.Table.from_batches array of values defined in schema as pyarrow.decimal128(12,2).
> >
> > Which values are in fact column of sqlAlchemy.sql.sqltypes.DECIMAL()="DECIMAL(precision=12,scale=2)"
values, which I map in schema like this pyarrow.decimal128(12,2).
> >
> > What could be a reason of issue?
> > Could it be that somehow declared decimal type of data is reduced to minimum value
which met in data? :/
> >
> > With best regards,
> > Dima Fomin

Mime
View raw message