From user-return-621-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sat Aug 15 21:07:41 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id D62AE18057A for ; Sat, 15 Aug 2020 23:07:41 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 01F5E124AC2 for ; Sat, 15 Aug 2020 21:07:41 +0000 (UTC) Received: (qmail 21997 invoked by uid 500); 15 Aug 2020 21:07:40 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 21983 invoked by uid 99); 15 Aug 2020 21:07:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2020 21:07:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D7226C01D5 for ; Sat, 15 Aug 2020 21:07:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.3 X-Spam-Level: X-Spam-Status: No, score=0.3 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id JBgRe_FY_tKS for ; Sat, 15 Aug 2020 21:07:38 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.166.173; helo=mail-il1-f173.google.com; envelope-from=wesmckinn@gmail.com; receiver= Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 74CD1BE369 for ; Sat, 15 Aug 2020 21:07:38 +0000 (UTC) Received: by mail-il1-f173.google.com with SMTP id f12so3766129ils.6 for ; Sat, 15 Aug 2020 14:07:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=AKSmGQmoLLMQl8fMgO+d5IgBTNPo5+uKAJhf3vg9YAQ=; b=duNRdo19d1c754BpdPajkqwXbRcE8Eeo5BGMh958AvPn+qu90eM9zYbaIm1XbKXv0m UN8wNqGPqXzvpdlFn6ApxyB7vDdNAXvMAdHbAmy3ZTUKQsNJ2OI4EW8Z8pCvwt4cBCOt /7rtynicTh2hWr7LV2/awczajz5p2pYYosY+mtfuXIzuSPL66U2teE3A/v3jYtEtVKQg 6TYwmQ8AOvwfqmqatLM5KE8OGX3Y8p2PUtfJB9lDBQhNzpdlsi385jRAqnFLM+OGeFqr Ox67xrlGfAV9DcZ7WwpWAs1ebPz+l6VKIl+0trX1++Z64ewEHHBsZ3KlZtlioU/ytgbf kfqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=AKSmGQmoLLMQl8fMgO+d5IgBTNPo5+uKAJhf3vg9YAQ=; b=tOZ5pz4KStj8z08NYJMmBm3TjsnNattn/u6t+YFA5pdjVm0EqrMIm0n1eKdwJomvlS 82NrCAfzuesNDE7aU2QJbGjdG6wnfB65XvrVp1C1drP4fMesR/vaUeKlr1QF9BxUxH9X Ya2WZCWQJIzA4Ddyc9Gx+SdY7s2vAQvaUCAKEiCG+wBvd+zWEImAPeUWlJouh6kWlnh5 QUlLU2H1v82o4TjG5BYeiSUj5P5hZkQVmrJSnFnuCrN0L3OW0HGFuR2XZ253jy3gWr1X Akh3qPwcFiq6Vjcfd6CeH/t/c2qUkf8oLNkL3l6tQMzOLvpZnwscVfKqSYcUUisWwhxW qBIw== X-Gm-Message-State: AOAM533E8F13U8tLwRoyronY2uyxlf3c0iiUarl+0rNPheJQyN/GJQCY XwPFoHKw+yq1qIa/NpxD8Hgdf8lN+BgN1imPEEftvcXMpFIhpy0/ X-Google-Smtp-Source: ABdhPJxVISZXqBoxdgS9L+u0Y1JxK446ZfL0xPK07et/U6j5Q9QFlD7NeTz2m39lTF6/AWG4hiQ2ZcdtqQScR+aHZcQ= X-Received: by 2002:a92:a157:: with SMTP id v84mr7489316ili.189.1597525657668; Sat, 15 Aug 2020 14:07:37 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Wes McKinney Date: Sat, 15 Aug 2020 16:07:01 -0500 Message-ID: Subject: Re: returning an arrow::Array from C++ to Python through pybind11 To: user@arrow.apache.org Content-Type: text/plain; charset="UTF-8" Are you using the arrow::py::wrap_array function? You can follow some other successful pybind11 projects that use the pyarrow C/C++ API. You have to also call the import_pyarrow() function https://github.com/blue-yonder/turbodbc/blob/0369d1329a0ea39982a4d8d169b8dd3f473e6689/cpp/turbodbc_arrow/Library/src/arrow_result_set.cpp#L338 On Fri, Aug 14, 2020 at 4:24 PM Max Grossman wrote: > > Hi all, > > I've written a C++ library that uses arrow for its in-memory data > structures. I'd like to also add some Python APIs on top of this > library using pybind11, so that I can grab pyarrow wrappers of my C++ > Arrow Arrays, convert them to numpy arrays, and then pass them in to > scikit-learn (or other Python libraries) without copying data around. > > As an example, on the C++ side I've got a 1D vector class that wraps > an arrow array and has a method to convert the arrow array into a > pyarrow array: > > PyObject* get_local_pyarrow_array() { > return arrow::py::wrap_array(std::dynamic_pointer_cast arrow::FixedSizeBinaryArray>(_arr)); > } > > I've got some pybind11 registration code that registers the class and > that method: > > py::class_>(m, "ShmemML1DD") > .def("get_local_pyarrow_array", > &ShmemML1D::get_local_pyarrow_array); > > And then I've got some Python code that calls this method (and which I > hope gets a pyarrow array as the return value): > > arr = dist_arr.get_local_pyarrow_array() > > Note that these are arrays that I'm constructing in C++ code and want > to expose to Python, so I don't already have a pre-existing pyarrow > instance to use. I'm trying to create a new one around my C++ arrays, > so that Python code can start manipulating those C++ arrays. > > When I build and run all this, I just get told "Unable to convert > function return value to a Python type!": > > Traceback (most recent call last): > File "/global/homes/j/jmg3/shmem_ml/example/python_wrapper.py", line > 15, in > random.rand(vec) > File "/global/homes/j/jmg3/shmem_ml/src/shmem_ml/random.py", line 8, in rand > arr = dist_arr.get_local_pyarrow_array() > TypeError: Unable to convert function return value to a Python type! > The signature was > (self: shmem_ml.core.ShmemML1DD) -> _object > Traceback (most recent call last): > File "/global/homes/j/jmg3/shmem_ml/example/python_wrapper.py", line > 15, in > random.rand(vec) > File "/global/homes/j/jmg3/shmem_ml/src/shmem_ml/random.py", line 8, in rand > arr = dist_arr.get_local_pyarrow_array() > TypeError: Unable to convert function return value to a Python type! > The signature was > (self: shmem_ml.core.ShmemML1DD) -> _object > > I'm new to pybind11, so I suspect this may not be a problem with my > arrow usage as much as it is with my pybind11 usage. I wanted to ask > if there's a better way to be doing this that's recommended for > pyarrow applications. It seems there are cython examples in the docs, > would the suggestion be to drop pybind11 and write a wrapper of my C++ > class in cython? > > Thanks for any suggestions, > > Max