drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@apache.org>
Subject Re: understanding groupCount & valueCount in repeated vectors
Date Fri, 27 Feb 2015 18:13:17 GMT
I woudl caution you that you avoid thinking about things in number of
records.  Complex repeated nested fields use these counts at their
respective context which isn't directly related to record counts (thus why
we initially chose not to call this getRecordCount).

On Fri, Feb 27, 2015 at 8:16 AM, Jason Altekruse <altekrusejason@gmail.com>
wrote:

> Hanifi,
>
> I think we should try to avoid using the word 'cell' to refer to elements
> within a single value. We often explain the concept of complex data in
> Drill by describing a list or map type being stored in a single database
> 'cell'. Overall I totally agree with the lack of clarity, I would advocate
> for something like getChildCount for the number of members below the lists,
> as current database language does not include hierarchies/nesting I think
> this is a safe naming convention.
>
> In response to Jacques comments, we might be at a loss with trying to unify
> the concepts of individual values in the case of scalar vectors and entire
> lists/nested structures with a simple name change. It might just be
> clearest to document the getValueCount method at the top level value vector
> interface to clearly state that it should match the number of records. Even
> beyond the issue of repeated confusion, this number also currently includes
> nulls, which some devs might find confusing if we don't document it.
>
> -Jason
>
> On Fri, Feb 27, 2015 at 6:24 AM, Jacques Nadeau <jacques@apache.org>
> wrote:
>
> > I think value is the problem word.  I'm not sure it is better for
> groupings
> > or cells in the case of repeated types.  What do they use in Parquet?
> >
> > I'd also like to see this proposal in the context of a larger proposed
> > design spec for that jira.
> > On Feb 26, 2015 5:52 PM, "Hanifi Gunes" <hgunes@maprtech.com> wrote:
> >
> > > Hey everyone,
> > >
> > > Scalar ValueVector(VV) types implement getValueCount method, which
> > returns
> > > the number of "value"s stored in the vector. I would expect the same be
> > > true for RepeatedVVs as well. However, getValueCount on repeated types
> > > report number of inner/sub-values stored and introduces another method
> > > called groupCount to report actual number of "value"s stored.
> > >
> > > This becomes really confusing and somewhat inconsistent (especially for
> > > RepeatedList) as one would expect #getValueCount should report the
> number
> > > of values regardless if the stored value type is nested or flat.
> > >
> > > As part of DRILL-2150, I am refactoring VVs so that getValueCount
> > > universally returns the number of values stored. Alongside, I plan to
> > > introduce a new method getCellCount that reports total number of
> > > sub-values/cells stored in a repeated vector.
> > >
> > > I'd like to probe if anyone has any concerns relating to this. Please
> let
> > > me know.
> > >
> > >
> > > Thanks.
> > > -Hanifi
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message