Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 209.85.217.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOKsKJVUvwWchs6_EUsFBw5PZQnRYhG7P_UzLzcMJ+sc9KGAEQ@mail.gmail.com>
References: 
 <CANZa=GuwDXAZUanE8mCDm-AE8ui5mT26HnBs-OwfXURBZOYTQA@mail.gmail.com>
	<CD7F4B28.204CB%doug.meil@explorysmedical.com>
	<CAOKsKJXC28xaD5i1UmCpn9p8=6kEend-i+0UuW9SM=0frGCWOw@mail.gmail.com>
	<CANZa=Gtb8=ytDsDKEfqo2ioEq4=aYxkx3bmiBY=0m650VA7eLQ@mail.gmail.com>
	<515A18CB.3050601@salesforce.com>
	<CANZa=GsnsGsWLjHxrZiZALQ0eqbE1EBn8T1z1OMYMbTUME9K7w@mail.gmail.com>
	<515A1CF6.5090804@salesforce.com>
	<CAOKsKJVUvwWchs6_EUsFBw5PZQnRYhG7P_UzLzcMJ+sc9KGAEQ@mail.gmail.com>
Date: Mon, 1 Apr 2013 17:10:41 -0700
Message-ID: 
 <CALte62xi5W7dkRryMevGix5xCPN09mYSWhfQxMrCrXWvZe-NDA@mail.gmail.com>
Subject: Re: HBase Types: Explicit Null Support
From: Ted Yu <yuzhihong@gmail.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=e89a8f23465966a05904d9559116

--e89a8f23465966a05904d9559116
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

bq. I create a dummy qualifier with a dummy value

For any single application, the above can be done.
For generic applications, how would we do this ?

Thanks


On Mon, Apr 1, 2013 at 5:07 PM, Matt Corgan <mcorgan@hotpads.com> wrote:

> I generally don't allow nulls in my composite row keys.  Does SQL allow
> nulls in the PK?  In the rare case I wanted to do that I might create a
> separate format called NullableCInt32 with 5 bytes where the first one
> determined null.  It's important to keep the pure types pure.
>
> I have lots of null *values* however, but they're represented by lack of =
a
> qualifier in the Put.  If a row has all null values, I create a dummy
> qualifier with a dummy value to make sure the row key gets inserted as it
> would in sql.
>
>
> On Mon, Apr 1, 2013 at 4:49 PM, James Taylor <jtaylor@salesforce.com>
> wrote:
>
> > On 04/01/2013 04:41 PM, Nick Dimiduk wrote:
> >
> >> On Mon, Apr 1, 2013 at 4:31 PM, James Taylor <jtaylor@salesforce.com>
> >> wrote:
> >>
> >>   From the SQL perspective, handling null is important.
> >>>
> >>
> >>  From your perspective, it is critical to support NULLs, even at the
> >> expense
> >> of fixed-width encodings at all or supporting representation of a full
> >> range of values. That is, you'd rather be able to represent NULL than
> >> -2^31?
> >>
> > We've been able to get away with supporting NULL through the absence of
> > the value rather than restricting the data range. We haven't had any pu=
sh
> > back on not allowing a fixed width nullable leading row key column. Sin=
ce
> > our variable length DECIMAL supports null and is a superset of the fixe=
d
> > width numeric types, users have a reasonable alternative.
> >
> > I'd rather not restrict the range of values, since it doesn't seem like
> > this would be necessary.
> >
> >
> >> On 04/01/2013 01:32 PM, Nick Dimiduk wrote:
> >>
> >>> Thanks for the thoughtful response (and code!).
> >>>>
> >>>> I'm thinking I will press forward with a base implementation that do=
es
> >>>> not
> >>>> support nulls. The idea is to provide an extensible set of interface=
s,
> >>>> so
> >>>> I
> >>>> think this will not box us into a corner later. That is, a mirroring
> >>>> package could be implemented that supports null values and accepts
> >>>> the relevant trade-offs.
> >>>>
> >>>> Thanks,
> >>>> Nick
> >>>>
> >>>> On Mon, Apr 1, 2013 at 12:26 PM, Matt Corgan <mcorgan@hotpads.com>
> >>>> wrote:
> >>>>
> >>>>   I spent some time this weekend extracting bits of our serializatio=
n
> >>>> code
> >>>>
> >>>>> to
> >>>>> a public github repo at http://github.com/hotpads/****data-tools<
> http://github.com/hotpads/**data-tools>
> >>>>> <http://github.com/**hotpads/data-tools<
> http://github.com/hotpads/data-tools>
> >>>>> >
> >>>>> .
> >>>>>    Contributions are welcome - i'm sure we all have this stuff layi=
ng
> >>>>> around.
> >>>>>
> >>>>> You can see I've bumped into the NULL problem in a few places:
> >>>>> *
> >>>>>
> >>>>> https://github.com/hotpads/****data-tools/blob/master/src/**<
> https://github.com/hotpads/**data-tools/blob/master/src/**>
> >>>>>
> main/java/com/hotpads/data/****primitive/lists/LongArrayList.****java<
> >>>>> https://github.com/**hotpads/data-tools/blob/**
> >>>>> master/src/main/java/com/**hotpads/data/primitive/lists/**
> >>>>> LongArrayList.java<
> https://github.com/hotpads/data-tools/blob/master/src/main/java/com/hotpa=
ds/data/primitive/lists/LongArrayList.java
> >
> >>>>> >
> >>>>> *
> >>>>>
> >>>>> https://github.com/hotpads/****data-tools/blob/master/src/**<
> https://github.com/hotpads/**data-tools/blob/master/src/**>
> >>>>> main/java/com/hotpads/data/****types/floats/DoubleByteTool.****java=
<
> >>>>> https://github.com/**hotpads/data-tools/blob/**
> >>>>> master/src/main/java/com/**hotpads/data/types/floats/**
> >>>>> DoubleByteTool.java<
> https://github.com/hotpads/data-tools/blob/master/src/main/java/com/hotpa=
ds/data/types/floats/DoubleByteTool.java
> >
> >>>>> >
> >>>>>
> >>>>> Looking back, I think my latest opinion on the topic is to reject
> >>>>> nullability as the rule since it can cause unexpected behavior and
> >>>>> confusion.  It's cleaner to provide a wrapper class (so both
> >>>>> LongArrayList
> >>>>> plus NullableLongArrayList) that explicitly defines the behavior, a=
nd
> >>>>> costs
> >>>>> a little more in performance.  If the user can't find a pre-made
> >>>>> wrapper
> >>>>> class, it's not very difficult for each user to provide their own
> >>>>> interpretation of null and check for it themselves.
> >>>>>
> >>>>> If you reject nullability, the question becomes what to do in
> >>>>> situations
> >>>>> where you're implementing existing interfaces that accept nullable
> >>>>> params.
> >>>>>    The LongArrayList above implements List<Long> which requires an
> >>>>> add(Long)
> >>>>> method.  In the above implementation I chose to swap nulls with
> >>>>> Long.MIN_VALUE, however I'm now thinking it best to force the user =
to
> >>>>> make
> >>>>> that swap and then throw IllegalArgumentException if they pass null=
.
> >>>>>
> >>>>>
> >>>>> On Mon, Apr 1, 2013 at 11:41 AM, Doug Meil <
> >>>>> doug.meil@explorysmedical.com
> >>>>>
> >>>>>  wrote:
> >>>>>> Hmmm=C5=A0 good question.
> >>>>>>
> >>>>>> I think that fixed width support is important for a great many
> rowkey
> >>>>>> constructs cases, so I'd rather see something like losing MIN_VALU=
E
> >>>>>> and
> >>>>>> keeping fixed width.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 4/1/13 2:00 PM, "Nick Dimiduk" <ndimiduk@gmail.com> wrote:
> >>>>>>
> >>>>>>   Heya,
> >>>>>>
> >>>>>>> Thinking about data types and serialization. I think null support
> is
> >>>>>>> an
> >>>>>>> important characteristic for the serialized representations,
> >>>>>>> especially
> >>>>>>> when considering the compound type. However, doing so in directly
> >>>>>>> incompatible with fixed-width representations for numerics. For
> >>>>>>>
> >>>>>>>  instance,
> >>>>>> if we want to have a fixed-width signed long stored on 8-bytes,
> where
> >>>>>> do
> >>>>>>
> >>>>>>> you put null? float and double types can cheat a little by foldin=
g
> >>>>>>> negative
> >>>>>>> and positive NaN's into a single representation (this isn't
> strictly
> >>>>>>> correct!), leaving a place to represent null. In the long example
> >>>>>>> case,
> >>>>>>> the
> >>>>>>> obvious choice is to reduce MAX_VALUE or increase MIN_VALUE by on=
e.
> >>>>>>> This
> >>>>>>> will allocate an additional encoding which can be used for null. =
My
> >>>>>>> experience working with scientific data, however, makes me wince =
at
> >>>>>>> the
> >>>>>>> idea.
> >>>>>>>
> >>>>>>> The variable-width encodings have it a little easier. There's
> already
> >>>>>>> enough going on that it's simpler to make room.
> >>>>>>>
> >>>>>>> Remember, the final goal is to support order-preserving
> >>>>>>> serialization.
> >>>>>>> This
> >>>>>>> imposes some limitations on our encoding strategies. For instance=
,
> >>>>>>> it's
> >>>>>>> not
> >>>>>>> enough to simply encode null, it really needs to be encoded as 0x=
00
> >>>>>>> so
> >>>>>>>
> >>>>>>>  as
> >>>>>> to sort lexicographically earlier than any other value.
> >>>>>>
> >>>>>>> What do you think? Any ideas, experiences, etc?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Nick
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >
>

--e89a8f23465966a05904d9559116--