Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of jianshi.huang@gmail.com
 designates 209.85.215.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAT7MkoVK3hFc56YPmSUgvik0-y5xqcORZ_7_f=dTbREEFnFJw@mail.gmail.com>
References: 
 <CACA1tW+roYVmjqaffZxczHKVZm0e9vEh3+zAkePo5rn0adfaOA@mail.gmail.com>
 <6AB6526B-AFA8-49F3-8CCA-64496B4E373B@gmail.com>
 <CACA1tWKXbTeZtpex-3fva1p1O2dxC5eHMOgm1E7FiH==pPbB4Q@mail.gmail.com>
 <CAAT7MkrC1RmaEbJ+Pe4v4iX9Xs3MGvWrp4csTWPY53bXwJP-kA@mail.gmail.com>
 <CACA1tW+uDkzGEqE4JRuOY3GxugytjSbVhBEQ+-TzbUGc2sq4Dg@mail.gmail.com>
 <CAAT7MkoVK3hFc56YPmSUgvik0-y5xqcORZ_7_f=dTbREEFnFJw@mail.gmail.com>
From: Jianshi Huang <jianshi.huang@gmail.com>
Date: Thu, 13 Nov 2014 17:27:43 +0800
Message-ID: 
 <CACA1tWK3EZQxifo43G3AjA4X0rgFJwc9uMzf0_KRpU3xhutPOA@mail.gmail.com>
Subject: Re: Storing JSON in HBase value cell, which serialization format is
 most compact?
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a1134849017714d0507ba21d5

--001a1134849017714d0507ba21d5
Content-Type: text/plain; charset=UTF-8

Thanks Ram,

How about Prefix Tree based encoding then? HBASE-4676
<https://issues.apache.org/jira/browse/HBASE-4676> says it's also possible
to do suffix tries? Then it could be a nice fit for JSON String (or any
long value where changes are small).

Maybe I should just flatten JSON to columns, hmm...what's the overhead for
a column?

Jianshi

On Thu, Nov 13, 2014 at 4:49 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> >>So is it possible to specify FASTDIFF for rowkey/column and DIFF for
> value
> cell?
> No that is not possible now. All the encoding is per KV only.
> But what you say is definitely worth trying.
>
> >>So would you recommend storing JSON flattened as many columns?
> May be yes.  But I have practically not used JSON formats so I may not be
> the best person to comment on this.
>
> Regards
> Ram
>
> On Thu, Nov 13, 2014 at 2:01 PM, Jianshi Huang <jianshi.huang@gmail.com>
> wrote:
>
> > Thanks Ram,
> >
> > So is it possible to specify FASTDIFF for rowkey/column and DIFF for
> value
> > cell?
> >
> > So would you recommend storing JSON flattened as many columns?
> >
> > Jianshi
> >
> > On Thu, Nov 13, 2014 at 2:08 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > Hi
> > >
> > > >> Since I'm storing
> > > historical data (snapshot data) and changes between adjacent value
> cells
> > > are relatively small.
> > >
> > > If the values are changing even if it is smaller the FASTDIFF will
> > rewrite
> > > the value part.  Only if there are exact matches then it would skip the
> > > value part. JFYI.
> > >
> > > Regards
> > > Ram
> > >
> > > On Thu, Nov 13, 2014 at 11:23 AM, Jianshi Huang <
> jianshi.huang@gmail.com
> > >
> > > wrote:
> > >
> > > > I thought FASTDIFF was only for rowkey and columns, great if it also
> > > works
> > > > in value cell.
> > > >
> > > > And thanks for the bjson link!
> > > >
> > > > Jianshi
> > > >
> > > > On Thu, Nov 13, 2014 at 1:18 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > > >
> > > > > There is FASTDIFF data block encoding.
> > > > >
> > > > > See also http://bjson.org/
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Nov 12, 2014, at 9:08 PM, Jianshi Huang <
> jianshi.huang@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm currently saving JSON in pure String format in the value cell
> > and
> > > > > > depends on HBase' block compression to reduce the overhead of
> JSON.
> > > > > >
> > > > > > I'm wondering if there's a more space efficient way to store
> JSON?
> > > > > > (there're lots of 0s and 1s, JSON String actually is an OK
> format)
> > > > > >
> > > > > > I want to keep the value as a Map since the schema of source data
> > > might
> > > > > > change over time.
> > > > > >
> > > > > > Also is there a DIFF based encoding for values? Since I'm storing
> > > > > > historical data (snapshot data) and changes between adjacent
> value
> > > > cells
> > > > > > are relatively small.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > --
> > > > > > Jianshi Huang
> > > > > >
> > > > > > LinkedIn: jianshi
> > > > > > Twitter: @jshuang
> > > > > > Github & Blog: http://huangjs.github.com/
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Jianshi Huang
> > > >
> > > > LinkedIn: jianshi
> > > > Twitter: @jshuang
> > > > Github & Blog: http://huangjs.github.com/
> > > >
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>


-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

--001a1134849017714d0507ba21d5--