arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Scheffers <...@youngbulls.nl>
Subject Re: Are Arrow, Flight and Plasma suitable for my use case?
Date Fri, 19 Mar 2021 14:02:37 GMT
Hi Matias,

A Tensor <https://en.wikipedia.org/wiki/Tensor>is a general name for a
n-dimensional array. The term has risen in popularity ever since TensorFlow
has gained traction as the most popular neural network framework in Python.

In the documentation tensors are not shown as an array data type, but
rather as an abstraction just like tables, see here
<https://arrow.apache.org/docs/python/api/tables.html>. Following this
discussion on github <https://github.com/apache/arrow/issues/4802>: Wes has
mentioned: "Tensors aren't part of the Arrow columnar format. Let's
continue the discussion on the mailing list or JIRA". Therefore, they are
not included as a data type and thus not usable as array format.

Regarding your question of the Plasma object store, from my perspective
Plasma is a generic byte object storage and thus not connected to the
discussion above.

Tom

Op vr 19 mrt. 2021 om 14:50 schreef Matias Guijarro <matias.guijarro@free.fr
>:

> Hi Fernando,
>
> Thanks for the information about Tensor (I wonder where the name comes
> from ?).
>
> Are tensors considered apart or is it possible to put them in a column
> like normal ?
>
> On 19/03/2021 10:20, Fernando Herrera wrote:
> >
> > However, I don't think the data stored in the tensor will be
> > compressed. It will be
> > orderly stored so you can share the tensors with other processes.
> >
>
> How do you share them with other processes ? Is it via Plasma ?
>
> (sorry for the noob questions - as said before I am investigating Arrow
> now, and
> essentially collecting information before trying anything)
>
> Thanks a lot,
> Matias.
>


-- 

Met vriendelijke groeten / Kind regards,

*Tom Scheffers*
+316 233 728 15


Young Bulls

Scheldestraat 21 - 1078 GD Amsterdam
<https://www.google.nl/maps/place/Scheldestraat+21,+1078+GD+Amsterdam/@52.3474466,4.889208,17z/data=!3m1!4b1!4m5!3m4!1s0x47c6098acd1783ed:0x649843bbcc11f7cd!8m2!3d52.3474466!4d4.8913967>

www.youngbulls.nl

Mime
View raw message