Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Received-SPF: pass (athena.apache.org: domain of altekrusejason@gmail.com
 designates 209.85.216.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKa9qD=8-HvwKaz+3vN1ck1RV8Af9BLW9DZDSwp7Zks5XbCxaQ@mail.gmail.com>
References: 
 <CANZQJAznveEZiDHkyzyCZ3Z7u7OTWwYS=HO6Q-Z+-xM0th-FNw@mail.gmail.com>
 <CAKa9qD=8-HvwKaz+3vN1ck1RV8Af9BLW9DZDSwp7Zks5XbCxaQ@mail.gmail.com>
From: Jason Altekruse <altekrusejason@gmail.com>
Date: Fri, 27 Feb 2015 08:16:21 -0800
Message-ID: 
 <CAMpYv7BN+qz3zYFkLtRLFuPaLGX1Q+cR7T8hAAzEGD0jqwimWA@mail.gmail.com>
Subject: Re: understanding groupCount & valueCount in repeated vectors
To: dev@drill.apache.org
Cc: drill <drill-dev@incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a1134a784a124460510143189

--001a1134a784a124460510143189
Content-Type: text/plain; charset=UTF-8

Hanifi,

I think we should try to avoid using the word 'cell' to refer to elements
within a single value. We often explain the concept of complex data in
Drill by describing a list or map type being stored in a single database
'cell'. Overall I totally agree with the lack of clarity, I would advocate
for something like getChildCount for the number of members below the lists,
as current database language does not include hierarchies/nesting I think
this is a safe naming convention.

In response to Jacques comments, we might be at a loss with trying to unify
the concepts of individual values in the case of scalar vectors and entire
lists/nested structures with a simple name change. It might just be
clearest to document the getValueCount method at the top level value vector
interface to clearly state that it should match the number of records. Even
beyond the issue of repeated confusion, this number also currently includes
nulls, which some devs might find confusing if we don't document it.

-Jason

On Fri, Feb 27, 2015 at 6:24 AM, Jacques Nadeau <jacques@apache.org> wrote:

> I think value is the problem word.  I'm not sure it is better for groupings
> or cells in the case of repeated types.  What do they use in Parquet?
>
> I'd also like to see this proposal in the context of a larger proposed
> design spec for that jira.
> On Feb 26, 2015 5:52 PM, "Hanifi Gunes" <hgunes@maprtech.com> wrote:
>
> > Hey everyone,
> >
> > Scalar ValueVector(VV) types implement getValueCount method, which
> returns
> > the number of "value"s stored in the vector. I would expect the same be
> > true for RepeatedVVs as well. However, getValueCount on repeated types
> > report number of inner/sub-values stored and introduces another method
> > called groupCount to report actual number of "value"s stored.
> >
> > This becomes really confusing and somewhat inconsistent (especially for
> > RepeatedList) as one would expect #getValueCount should report the number
> > of values regardless if the stored value type is nested or flat.
> >
> > As part of DRILL-2150, I am refactoring VVs so that getValueCount
> > universally returns the number of values stored. Alongside, I plan to
> > introduce a new method getCellCount that reports total number of
> > sub-values/cells stored in a repeated vector.
> >
> > I'd like to probe if anyone has any concerns relating to this. Please let
> > me know.
> >
> >
> > Thanks.
> > -Hanifi
> >
>

--001a1134a784a124460510143189--