Mailing-List: contact hive-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hive-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of heyongqiangict@gmail.com
 designates 74.125.82.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=IWWhDx7DpUPBXqZpZnAeX5ZKFzD92fq/dJC7589E/rMwTIeJVQJdLLNAALcWmqQADX
         2F/gtBuhnxkg8cq8biS1zRXf3hgreDO1ld4sDX6YnnBMiGyPXarLpzodSmsAdRo3iPsa
         gC/gPgZsiPDgNTZm6BTVYSzdCuebZa2Mp0OEQ=
MIME-Version: 1.0
In-Reply-To: <C1543142-F26C-4C9F-B0EB-5C783555C0A9@facebook.com>
References: 
 <362A2DDD005E584BAB2BCE3E2FE0E12212D7845378@SP2-EX07VS02.ds.corp.yahoo.com>
	<C1543142-F26C-4C9F-B0EB-5C783555C0A9@facebook.com>
Date: Mon, 9 Aug 2010 13:07:17 -0700
Message-ID: <AANLkTi=6Pbdvnvn6o0KdvvDSioJ56kU_=v9FV2SyxB90@mail.gmail.com>
Subject: Re: How are nulls represented in data?
From: yongqiang he <heyongqiangict@gmail.com>
To: hive-user@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Yes. In LazySimpleSerde/SequenceFile/TextFile, "\N" is used as NULL.
(It is a table property: serialization.null.format)

In ColumnSerDe/RCFile, there is no NULL stored. (zero byte, column
byte length is zero).
But RCFile/ColumnarSerde also use this property when do serializing to
determine if a column is a null or not. ( This is unavoidable because
client can only pass a string to serde and let serde serialize it.
need some special charater to represent NULL).

On Mon, Aug 9, 2010 at 11:46 AM, Ning Zhang <nzhang@facebook.com> wrote:
> How it is serialized/deserialized is determined by specific serde.=A0NULL=
 is
> serialized as \N by SimpleLazySerDe (default serde for text).=A0RCFile
> (ColumnarSerDe) uses the same default parameters as LazySimpleSerDe.
> Unless I missed something, NULL serialization/deserialization is type
> independent (at least in LazySimpleSerDe).
> On Aug 9, 2010, at 9:42 AM, Pradeep Kamath wrote:
>
> Hi,
> =A0=A0 What value does hive expect in the data for a column to be treated=
 as
> null? I tried some permutations on a text data based table but couldn=92t
> figure out what the correct representation was. I tried empty string, the
> string NULL and the string null for a string column and in all three case=
s
> the =93is null=94 operator returned false.
>
> A couple of related questions:
> =A0- Does the representation of null depend on the type of the column =96=
 is it
> different for string Vs non-string columns?
> =A0- Is the representation of null different for different storage format=
s =96
> text Vs RCFile Vs SequenceFile =96 I am particularly interested in text a=
nd
> RCFile.
>
> Thanks in advance,
>
> Pradeep
>