Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAAg3a2pVrZR1U8tLPx7wd-q_pdG1fY-kLuuZpd4w_z7Re234ag@mail.gmail.com>
References: 
 <CA+nge+r8cZQv1bLvN5=pWCia8AkQ6dmtwBhOzREVanwOfpddfA@mail.gmail.com>
 <CALte62z-w2n5RZvUO-7r5Y-AFr2rJHCjih4zuc99dQETyzpvmA@mail.gmail.com>
 <CAAg3a2pVrZR1U8tLPx7wd-q_pdG1fY-kLuuZpd4w_z7Re234ag@mail.gmail.com>
From: Gaurav Agarwal <gaurav@arkin.net>
Date: Thu, 24 Sep 2015 00:10:37 +0530
Message-ID: 
 <CA+nge+r-_5nT+He7=qZyjqaC+W-2dUAGs_MRVrV66bH6mgNLPw@mail.gmail.com>
Subject: Re: Large number of column qualifiers
To: Hbase Users <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a113ebb32b88c7105206e758b

--001a113ebb32b88c7105206e758b
Content-Type: text/plain; charset=UTF-8

Thanks Vlad. Could you please point me the KV size setting (default 1MB)?
Just to make sure that I understand correct - the following comment is
incorrect in Cell.java:

 /**
   * Contiguous raw bytes that may start at any index in the containing
array. Max length is
   * Short.MAX_VALUE which is 32,767 bytes.
   * @return The array containing the qualifier bytes.
   */
  byte[] getQualifierArray();

On Wed, Sep 23, 2015 at 11:43 PM, Vladimir Rodionov <vladrodionov@gmail.com>
wrote:

> Check KeyValue class (Cell's implementation). getQualifierArray() returns
> kv's backing array. There is no SHORT limit on a size of this array, but
> there are other limits in  HBase - maximum KV size, for example, which is
> configurable, but, by default, is 1MB. Having 50K qualifiers is a bad idea.
> Consider redesigning your data model and use rowkey instead.
>
> -Vlad
>
> On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Please take a look at HBASE-11544 which is in hbase 1.1
> >
> > Cheers
> >
> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal <gaurav@arkin.net>
> wrote:
> >
> > > Hi All,
> > >
> > > I have Column Family with very large number of column qualifiers (>
> > > 50,000). Each column qualifier is 8 bytes long. The problem is the
> when I
> > > do a scan operation to fetch some rows, the client side Cell object
> does
> > > not have enough space allocated in it to hold all the columnQaulifiers
> > for
> > > a given row and hence I cannot read all the columns back for a given
> row.
> > >
> > > Please see the code snippet that I am using:
> > >
> > >  final ResultScanner rs = htable.getScanner(scan);
> > >  for (Result row = rs.next(); row != null; row = rs.next()) {
> > >     final Cell[] cells = row.rawCells();
> > >     if (cells != null) {
> > >         for (final Cell cell : cells) {
> > >             final long c = Bytes.toLong(
> > >                     *cell.getQualifierArray()*,
> > cell.getQualifierOffset(),
> > > cell.getQualifierLength());
> > >             final long v = Bytes.toLong(cell.getValueArray(),
> > > cell.getValueOffset());
> > >             points.put(c, v);
> > >         }
> > >     }
> > > }
> > >
> > > The cell.getQualifierArray() method says that it's 'Max length is
> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
> > > 4,000 columnQualfiers.
> > >
> > > Is there an alternate API that I should be using or am I missing some
> > > setting here? Note that in worst case I need to read all the
> > > columnQualifiers in a row and I may or may not know a subset to fetch
> in
> > > advance.
> > >
> > > Even if this is not possible in a single call, is there a way to cursor
> > > through the columnQualifiers?
> > >
> > > I am presently using Hbase 0.96 client but can switch to Hbase 1.x if
> > there
> > > is an API in the newer version.
> > >
> > > --cheers, gaurav
> > >
> > > --
> > > --cheers, gaurav
> > >
> >
>


-- 
--cheers, gaurav

--001a113ebb32b88c7105206e758b--