Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CA+nge+r-_5nT+He7=qZyjqaC+W-2dUAGs_MRVrV66bH6mgNLPw@mail.gmail.com>
References: 
 <CA+nge+r8cZQv1bLvN5=pWCia8AkQ6dmtwBhOzREVanwOfpddfA@mail.gmail.com>
 <CALte62z-w2n5RZvUO-7r5Y-AFr2rJHCjih4zuc99dQETyzpvmA@mail.gmail.com>
 <CAAg3a2pVrZR1U8tLPx7wd-q_pdG1fY-kLuuZpd4w_z7Re234ag@mail.gmail.com>
 <CA+nge+r-_5nT+He7=qZyjqaC+W-2dUAGs_MRVrV66bH6mgNLPw@mail.gmail.com>
From: Gaurav Agarwal <gaurav@arkin.net>
Date: Thu, 24 Sep 2015 00:15:30 +0530
Message-ID: 
 <CA+nge+rWB8hk648Kt6kQ+JnoVsCp3zpf30NF1OmHmJZmB-=ySg@mail.gmail.com>
Subject: Re: Large number of column qualifiers
To: Hbase Users <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7bd75c3a3184c705206e876e

--047d7bd75c3a3184c705206e876e
Content-Type: text/plain; charset=UTF-8

Thanks Vlad. Could you please point me the KV size setting (default 1MB)?
Just to make sure that I understand correct, are you suggesting that the
following comment is incorrect in Cell.java?

 /**
   * Contiguous raw bytes that may start at any index in the containing
array. Max length is
   * Short.MAX_VALUE which is 32,767 bytes.
   * @return The array containing the qualifier bytes.
   */
  byte[] getQualifierArray();

On Thu, Sep 24, 2015 at 12:10 AM, Gaurav Agarwal <gaurav@arkin.net> wrote:

> Thanks Vlad. Could you please point me the KV size setting (default 1MB)?
> Just to make sure that I understand correct - the following comment is
> incorrect in Cell.java:
>
>  /**
>    * Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>    * Short.MAX_VALUE which is 32,767 bytes.
>    * @return The array containing the qualifier bytes.
>    */
>   byte[] getQualifierArray();
>
> On Wed, Sep 23, 2015 at 11:43 PM, Vladimir Rodionov <
> vladrodionov@gmail.com> wrote:
>
>> Check KeyValue class (Cell's implementation). getQualifierArray() returns
>> kv's backing array. There is no SHORT limit on a size of this array, but
>> there are other limits in  HBase - maximum KV size, for example, which is
>> configurable, but, by default, is 1MB. Having 50K qualifiers is a bad
>> idea.
>> Consider redesigning your data model and use rowkey instead.
>>
>> -Vlad
>>
>> On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> > Please take a look at HBASE-11544 which is in hbase 1.1
>> >
>> > Cheers
>> >
>> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal <gaurav@arkin.net>
>> wrote:
>> >
>> > > Hi All,
>> > >
>> > > I have Column Family with very large number of column qualifiers (>
>> > > 50,000). Each column qualifier is 8 bytes long. The problem is the
>> when I
>> > > do a scan operation to fetch some rows, the client side Cell object
>> does
>> > > not have enough space allocated in it to hold all the columnQaulifiers
>> > for
>> > > a given row and hence I cannot read all the columns back for a given
>> row.
>> > >
>> > > Please see the code snippet that I am using:
>> > >
>> > >  final ResultScanner rs = htable.getScanner(scan);
>> > >  for (Result row = rs.next(); row != null; row = rs.next()) {
>> > >     final Cell[] cells = row.rawCells();
>> > >     if (cells != null) {
>> > >         for (final Cell cell : cells) {
>> > >             final long c = Bytes.toLong(
>> > >                     *cell.getQualifierArray()*,
>> > cell.getQualifierOffset(),
>> > > cell.getQualifierLength());
>> > >             final long v = Bytes.toLong(cell.getValueArray(),
>> > > cell.getValueOffset());
>> > >             points.put(c, v);
>> > >         }
>> > >     }
>> > > }
>> > >
>> > > The cell.getQualifierArray() method says that it's 'Max length is
>> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
>> > > 4,000 columnQualfiers.
>> > >
>> > > Is there an alternate API that I should be using or am I missing some
>> > > setting here? Note that in worst case I need to read all the
>> > > columnQualifiers in a row and I may or may not know a subset to fetch
>> in
>> > > advance.
>> > >
>> > > Even if this is not possible in a single call, is there a way to
>> cursor
>> > > through the columnQualifiers?
>> > >
>> > > I am presently using Hbase 0.96 client but can switch to Hbase 1.x if
>> > there
>> > > is an API in the newer version.
>> > >
>> > > --cheers, gaurav
>> > >
>> > > --
>> > > --cheers, gaurav
>> > >
>> >
>>
>
>
>
> --
> --cheers, gaurav
>


-- 
--cheers, gaurav

--047d7bd75c3a3184c705206e876e--