From user-return-1219-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed May 5 05:42:09 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 81D6E18062C for ; Wed, 5 May 2021 07:42:09 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 9860060B09 for ; Wed, 5 May 2021 05:42:08 +0000 (UTC) Received: (qmail 9498 invoked by uid 500); 5 May 2021 05:42:06 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 9487 invoked by uid 99); 5 May 2021 05:42:05 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2021 05:42:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id D69DE1FF3A1 for ; Wed, 5 May 2021 05:42:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id 3hHOu8JB7vhg for ; Wed, 5 May 2021 05:42:04 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::530; helo=mail-ed1-x530.google.com; envelope-from=emkornfield@gmail.com; receiver= Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 6DBE97FD9C for ; Wed, 5 May 2021 05:42:04 +0000 (UTC) Received: by mail-ed1-x530.google.com with SMTP id j28so459352edy.9 for ; Tue, 04 May 2021 22:42:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to; bh=Gg1r/HzLvY2IFCEm4anLOzVR9eGvv8IOpsuHYQPnWlc=; b=X0Npau4NwmYXS2DCXU+XE9ux68SWlMksVN32YZOBYb6/jrX626FhVL0bQOxedhscFP Rz6viYKsxqIjO9zFsn2y2aYgHWywvQ76cK72Ydy1pUxO06my5g2anSIZQ73XALYaiPpP WX1JtZgPE/WvFnEqPvK8twhFYaMF5vVznPrPmMMruoQ1cckZeRZRZYeXaNQnOYbY/ePB Yd7fib1cp8PykNP/L20gaF8BnjRL+N1U//ard6638UTxl3moybXi3JCBWx5T3AYOr8ZR KgKH77DVU0piq50VD/6vUMwkxsb3bYu6T56pkxZagPHbvzbcsd4XJb71BgPwpLAfcYSy HWaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to; bh=Gg1r/HzLvY2IFCEm4anLOzVR9eGvv8IOpsuHYQPnWlc=; b=Rw13Qi9N79kuAopjyurHIIzdIHW1FS+zCJCip71hXWh089C20ZcYcC7irtjY9XSUaU 3eRsgVxLgmKmPw8TFzp2YB8AIc7One97L+rmwRZoX5OPsyHmWyaxBryPCMy+MqBifJQW ++iGme47VZdz88TalPYD03iWwnws3F29hvx08yea7+xU2TSTp0DmGoFDYUICd66h0mPP gsdTQK8xxV4RgFIrdqXw0Pe6CuRHCzsmEF9Y+ZSPGusp3NqH0/XEqtz80PzhnQ9FN07v MWEifPeZ0l1JtD3MiUlmoBFkn6PoZQkWkio1xnHZxGCh545+vsDqg5YCIgGriqJCkffb 86UA== X-Gm-Message-State: AOAM530gdsGlP+T9BdTILfe3I5p9KvOk8woOjp7h+yGnKHZfwg921too f2X8uqA3rSshF+dczhtW0am9H4QgSJZ1wwRUaqE1SvIK X-Google-Smtp-Source: ABdhPJwDcDzo81niKAV4iPUb857mtYKhKsDULHA2lI+bcFMHQuWlrdAy9mXagHD3pSArYhdG+4ETRyQRrDI4wrizxCg= X-Received: by 2002:aa7:c7d5:: with SMTP id o21mr30526736eds.166.1620193323733; Tue, 04 May 2021 22:42:03 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: Reply-To: emkornfield@gmail.com From: Micah Kornfield Date: Tue, 4 May 2021 22:41:51 -0700 Message-ID: Subject: Re: [Java JDBC adapter] non-nullable fields? To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="00000000000087584105c18ea7bc" --00000000000087584105c18ea7bc Content-Type: text/plain; charset="UTF-8" I would need to look further, but I thought we handled null vs not null. At least I thought we had specialized conversion code to avoid branches. If this isn't the case it seems reasonable to contribute a path. On Tue, May 4, 2021 at 3:39 AM Joris Peeters wrote: > I'm looking to use the Java JDBC adapter for loading tables from SQL > Server into Arrow record batches. > > At first glance the Arrow JDBC adapter seems to work well but, unless I'm > mistaken, it simply makes every vector nullable, irrespective of whether > the corresponding SQL column is nullable or not. > > I think the line > > final FieldType fieldType = new FieldType(true, arrowType, /* dictionary > encoding */ null, metadata); > > in > https://github.com/apache/arrow/blob/master/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java#L158 > might be the cause here. > > Is my interpretation correct, or am I missing a setting of sorts? If > indeed correct, is there a fundamental reason the NULL-ness is not > transferred, or is this something I could contribute in a PR? (which I'd be > happy to) I guess it's just a matter of inspecting the result metadata. > > Cheers, > -J > --00000000000087584105c18ea7bc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I would need to look further, but I thought we handled nul= l vs not null.=C2=A0 At least I thought we had specialized conversion code = to avoid branches.=C2=A0 If this isn't the case it seems reasonable to = contribute a path.

On Tue, May 4, 2021 at 3:39 AM Joris Peeters <joris.mg.peeters@gmail.com> w= rote:
I'm looking to use the Java JDBC adapter for loading tables fr= om SQL Server into Arrow record batches.=C2=A0

At first = glance the Arrow JDBC adapter seems to work well but, unless I'm mistak= en, it simply makes every vector nullable, irrespective of whether the corr= esponding SQL column is nullable or not.=C2=A0

I t= hink the line=C2=A0

final FieldType fieldType =3D = new FieldType(true, arrowType, /* dictionary encoding */ null, metadata);


Is my interpretation correct, or am I missing a setting of sorts? If = indeed correct, is there a fundamental reason the NULL-ness is not transfer= red, or is this something I could contribute in a PR? (which I'd be hap= py to) I guess it's just a matter of inspecting the result metadata.

Cheers,
-J
--00000000000087584105c18ea7bc--