From user-return-1222-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Thu May 6 04:53:56 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 952EE18062C for ; Thu, 6 May 2021 06:53:56 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id B265C60EE8 for ; Thu, 6 May 2021 04:53:55 +0000 (UTC) Received: (qmail 64827 invoked by uid 500); 6 May 2021 04:53:54 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 64817 invoked by uid 99); 6 May 2021 04:53:54 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 May 2021 04:53:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 7F84DC0104 for ; Thu, 6 May 2021 04:53:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 0.249 X-Spam-Level: X-Spam-Status: No, score=0.249 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id Xy6slVzLF2g6 for ; Thu, 6 May 2021 04:53:52 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.219.175; helo=mail-yb1-f175.google.com; envelope-from=liya.fan03@gmail.com; receiver= Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 69F3CBD13B for ; Thu, 6 May 2021 04:53:52 +0000 (UTC) Received: by mail-yb1-f175.google.com with SMTP id h202so5706222ybg.11 for ; Wed, 05 May 2021 21:53:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=njzQgv/j1BgTG4YMLGKDiT3AQ64Vxov9udZBkBKS+oM=; b=eXB2zjIErLszGMrhq802y0XeR1NiOejBHQXCBsm3s9++4gyIwVb7vAg4ctNEWhz3kc uoqhJTFqbf/3l7bUdaDVHY6k6Xmk5QFy0btQv7CmZyy+VloC4HhyQ4G0MvVTT5/TebrP jN8eXsMQekXD3ACzIZj8PHPEvsdJqZeO4OcJM5QN2wYAREAXsswygfi6QOm80rpybWyf cUUxnOcu7lvhAjUHrAmBE8Az3fWc1Yt7zM8TUILhnnZVehMoXE8f56KkYEr5Orh7w62r FlC5InVhKMyBs/Un8SRQYbAvn5ugMzJokqzo4cz7XDpLc7V3m8TajY849RjlL9YIErir HEMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=njzQgv/j1BgTG4YMLGKDiT3AQ64Vxov9udZBkBKS+oM=; b=gKjj+10FmIET2wuKtqZGKNbPSm/P/HLJmOU7VFdUeQ4UoShqnj/pXlBNzn+Cyyso+k Z71wWGNt6wM1yZOuusZIZjbfBvrBv459mmNldfuVdQfvBOH3RBsrRct9ZuMAUPQ7Walc /Xz55CYXVzz6wW2KDAMzc7ojClr80UDQdt65NwoEf3/5GRC9lYFj5TMSskpHcO+vmRA6 QEmn7n7T4E/v2w8S/kHXMAk10gKNWvNEordN4eAbhdl4QGAoVAu6IwOGPg/i93+BEnQG XGL9ve0TiU8ZACUqIVjb4RV9jx45ir+J47uQBsVmCqqsqw2l3v8u8pxKECkllFVjXbQQ tQlg== X-Gm-Message-State: AOAM5319jbYMuaM7wxrKPrR/zhi02Lrt38HvXG5Ji2QXbymPW14vSKJl 6UhNu1QbW/WOMPu4YJagxBjCtpOKCGd1wp+Zzs/ldSnhi5k= X-Google-Smtp-Source: ABdhPJyKerHNWZoR7pt98q7Yw3nx7+p723gNq2SqH72+BSKz0sTjrTbwvH4O01l4ptiWpUGnyGI3qeLX/+WJKcHyy3I= X-Received: by 2002:a25:6003:: with SMTP id u3mr3248688ybb.96.1620276825747; Wed, 05 May 2021 21:53:45 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Fan Liya Date: Thu, 6 May 2021 12:52:47 +0800 Message-ID: Subject: Re: [Java JDBC adapter] non-nullable fields? To: user@arrow.apache.org, Micah Kornfield Content-Type: multipart/alternative; boundary="000000000000a2f42905c1a2181e" --000000000000a2f42905c1a2181e Content-Type: text/plain; charset="UTF-8" Hi Joris, Thanks for reporting the problem. We make use of the nullable information in ArrowVectorIterator#initialize. (Details can be found in https://github.com/apache/arrow/blob/master/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/ArrowVectorIterator.java#L73 ) Please note that the ArrowVectorIterator is our encouraged way of using the JDBC adapter. Best, Liya Fan On Wed, May 5, 2021 at 1:42 PM Micah Kornfield wrote: > I would need to look further, but I thought we handled null vs not null. > At least I thought we had specialized conversion code to avoid branches. > If this isn't the case it seems reasonable to contribute a path. > > On Tue, May 4, 2021 at 3:39 AM Joris Peeters > wrote: > >> I'm looking to use the Java JDBC adapter for loading tables from SQL >> Server into Arrow record batches. >> >> At first glance the Arrow JDBC adapter seems to work well but, unless I'm >> mistaken, it simply makes every vector nullable, irrespective of whether >> the corresponding SQL column is nullable or not. >> >> I think the line >> >> final FieldType fieldType = new FieldType(true, arrowType, /* dictionary >> encoding */ null, metadata); >> >> in >> https://github.com/apache/arrow/blob/master/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java#L158 >> might be the cause here. >> >> Is my interpretation correct, or am I missing a setting of sorts? If >> indeed correct, is there a fundamental reason the NULL-ness is not >> transferred, or is this something I could contribute in a PR? (which I'd be >> happy to) I guess it's just a matter of inspecting the result metadata. >> >> Cheers, >> -J >> > --000000000000a2f42905c1a2181e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Joris,

Thanks for reporting the prob= lem.

We make use of the nullable information in=C2= =A0ArrowVectorIterator#initialize. (Details can be found in https://github.com= /apache/arrow/blob/master/java/adapter/jdbc/src/main/java/org/apache/arrow/= adapter/jdbc/ArrowVectorIterator.java#L73)

Ple= ase note that the=C2=A0 ArrowVectorIterator is our encouraged way of using the JDBC adapter.
<= div>
Best,
Liya Fan


<= div class=3D"gmail_quote">
On Wed, May= 5, 2021 at 1:42 PM Micah Kornfield <emkornfield@gmail.com> wrote:
I would need to look further, b= ut I thought we handled null vs not null.=C2=A0 At least I thought we had s= pecialized conversion code to avoid branches.=C2=A0 If this isn't the c= ase it seems reasonable to contribute a path.

On Tue, May 4, 2021 at 3:39 AM= Joris Peeters <joris.mg.peeters@gmail.com> wrote:
I'm looking to use t= he Java JDBC adapter for loading tables from SQL Server into Arrow record b= atches.=C2=A0

At first glance the Arrow JDBC adapter see= ms to work well but, unless I'm mistaken, it simply makes every vector = nullable, irrespective of whether the corresponding SQL column is nullable = or not.=C2=A0

I think the line=C2=A0
final FieldType fieldType =3D new FieldType(true, arrowType, /*= dictionary encoding */ null, metadata);


Is my interpretation correc= t, or am I missing a setting of sorts? If indeed correct, is there a fundam= ental reason the NULL-ness is not transferred, or is this something I could= contribute in a PR? (which I'd be happy to) I guess it's just a ma= tter of inspecting the result metadata.

Cheers,
-J
--000000000000a2f42905c1a2181e--