From user-return-979-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Thu Feb 11 17:40:36 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 8703C18065D for ; Thu, 11 Feb 2021 18:40:36 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id BF61B44A42 for ; Thu, 11 Feb 2021 17:40:35 +0000 (UTC) Received: (qmail 61301 invoked by uid 500); 11 Feb 2021 17:40:35 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 61291 invoked by uid 99); 11 Feb 2021 17:40:35 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2021 17:40:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 93BB5C0116 for ; Thu, 11 Feb 2021 17:40:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: -0.202 X-Spam-Level: X-Spam-Status: No, score=-0.202 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id S_ykD0O-CFkF for ; Thu, 11 Feb 2021 17:40:34 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.43; helo=mail-lf1-f43.google.com; envelope-from=wesmckinn@gmail.com; receiver= Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 59975BCC6B for ; Thu, 11 Feb 2021 17:40:33 +0000 (UTC) Received: by mail-lf1-f43.google.com with SMTP id m22so9322116lfg.5 for ; Thu, 11 Feb 2021 09:40:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=AjLxtGzYkQmnUwQ9IuuikikAXDUO9gHM3eh+LmRSRUU=; b=pu5RESNdS7NMRWNVOh10aVCNjifPMwS6azDHbaBIHkTbNwQfgDrC2eQoCtiizbFvN1 7gjgOZ02TbA/q6gFx7nVjaq52EcDXtn8c+K5+teEj9oQYijfCrRuKZeEVEOyI5bWTcTc B6HW8FUWU1QLtzsGs5Bb1JVXRVsiL62dxBZrdYKbpQQ/GiyV2Pz6PeQ7RrD/VuUiquDT Zmk3uAcBI2c6MoxNEdcJNYZH/WTLxJl/Ghi7k5kP4i85JAvH6aF4VPTajBTqrRzV5Mla OfkqCsVwbF5PSCyRq1y0NbK62v17FtB+RalD3bVMDZ/vNu9KBUzpiNvscF4DnpCBmt0k J08Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=AjLxtGzYkQmnUwQ9IuuikikAXDUO9gHM3eh+LmRSRUU=; b=ZVZQfftc9414AVrVFkSHqvij8V0bbnj2K+MjPmT+u1SLeeiP4BqEQFUNrXhh2CGDUw cwYqpuEmaWtKB9b7vYJfS/xYbTNu3HJYq9AeH2d+0D7AMsbCq5HYN+CaLd/dxx3mm/zA /ZuQW1kfj5F8v0I9TQWgWXmdlYzAFIdOgeHc8rXcb7zburMgRBH9kYt8o4CaLoNf1DaB QITdMDIh2QmDb1Xr3SQYaBxQKoCZC3ZSXatOQ19lFP3OaKyot33vTKYVhnCKuw+1RSvq 9b//g89CUAqnKHM5qh6jDrAJdIljFdedPSBLyY5ezObSds4SxV3L0AKkayFyMIFxM7cJ 4abA== X-Gm-Message-State: AOAM532hXSEVFx23rrLlzQZ/+BZLXDMDljHEgVE4nQ1NraraWKa759BK /Zbos1KHoCKS67P1Sb8Qo1tVgVptZVqqtA8H0J0wi7hHmQ0= X-Google-Smtp-Source: ABdhPJw1WAxcgePwuH+3OtNEQjLj6MqWRX0cG2HWSGDvXvM6U9ORXNj7GWW/Z0++igtB2mEU1Zuhg1AlTB13anD9q+8= X-Received: by 2002:a05:6512:2214:: with SMTP id h20mr596949lfu.81.1613065231913; Thu, 11 Feb 2021 09:40:31 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Wes McKinney Date: Thu, 11 Feb 2021 11:39:54 -0600 Message-ID: Subject: Re: pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values To: user@arrow.apache.org Content-Type: text/plain; charset="UTF-8" We simply don't have conversions nor type inference implemented for inner elements with dimension greater than 1. You're welcome to propose this as a new feature / enhancement by opening a Jira issue. On Thu, Feb 11, 2021 at 10:53 AM Bhavitvya Malik wrote: > > Sure, here it is: > > data = np.zeros((10,8), dtype=np.uint8) > out = pa.array(list(data)) > out.type # ListType(list) > > data = np.zeros((3,4,6), dtype=np.uint8) > out = pa.array(list(data)) # Throws error ArrowInvalid: Can only convert 1-dimensional array values > > Even though it's working on 2D numpy arrays perfectly, it doesn't work on N-Dimensional numpy arrays (where N > 2). Why is it so? > > > On Thu, 11 Feb 2021 at 21:18, Wes McKinney wrote: >> >> Can you provide more detail about what you are trying? You've showed >> some exception here but haven't showed the exact code that results in >> those exceptions >> >> On Thu, Feb 11, 2021 at 4:34 AM Bhavitvya Malik >> wrote: >> > >> > Hi, >> > It's a follow up question for #9462. Rewriting the issue here: >> > >> >> I came to know that pyarrow has this limitation of not storing N-dimensional array. After looking into this issue, I decided to represent a N-dimensional array as a list of arrays i.e. >> >> data = np.zeros((5, 3), dtype=np.uint8) >> >> data = list(data) >> >> inorder to preserve the dtype but when it comes to typecasting and writing it into array (from list) pyarrow.array(data, type=type) it gives the following error: >> >> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type numpy.ndarray: tried to convert to int >> >> Is there any way to avoid this issue? I just want to preserve the dtype from numpy array before converting it to list so that while writing it to pyarrow array format I can recognise its dtype and subsequently write it in that numpy dtype format. >> > >> > >> > I tried it with a 3D numpy array and it gave me this error even though it's working fine with 2D numpy arrays. Can you please look into this? >> > >> > pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values >> > >> > >> > My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0 too >> > >> > >> > >> > Thanks, >> > Bhavitvya