Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CALte62z-w2n5RZvUO-7r5Y-AFr2rJHCjih4zuc99dQETyzpvmA@mail.gmail.com>
References: 
 <CA+nge+r8cZQv1bLvN5=pWCia8AkQ6dmtwBhOzREVanwOfpddfA@mail.gmail.com>
	<CALte62z-w2n5RZvUO-7r5Y-AFr2rJHCjih4zuc99dQETyzpvmA@mail.gmail.com>
Date: Wed, 23 Sep 2015 11:13:31 -0700
Message-ID: 
 <CAAg3a2pVrZR1U8tLPx7wd-q_pdG1fY-kLuuZpd4w_z7Re234ag@mail.gmail.com>
Subject: Re: Large number of column qualifiers
From: Vladimir Rodionov <vladrodionov@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7bb04db076830005206e12b1

--047d7bb04db076830005206e12b1
Content-Type: text/plain; charset=UTF-8

Check KeyValue class (Cell's implementation). getQualifierArray() returns
kv's backing array. There is no SHORT limit on a size of this array, but
there are other limits in  HBase - maximum KV size, for example, which is
configurable, but, by default, is 1MB. Having 50K qualifiers is a bad idea.
Consider redesigning your data model and use rowkey instead.

-Vlad

On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Please take a look at HBASE-11544 which is in hbase 1.1
>
> Cheers
>
> On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal <gaurav@arkin.net> wrote:
>
> > Hi All,
> >
> > I have Column Family with very large number of column qualifiers (>
> > 50,000). Each column qualifier is 8 bytes long. The problem is the when I
> > do a scan operation to fetch some rows, the client side Cell object does
> > not have enough space allocated in it to hold all the columnQaulifiers
> for
> > a given row and hence I cannot read all the columns back for a given row.
> >
> > Please see the code snippet that I am using:
> >
> >  final ResultScanner rs = htable.getScanner(scan);
> >  for (Result row = rs.next(); row != null; row = rs.next()) {
> >     final Cell[] cells = row.rawCells();
> >     if (cells != null) {
> >         for (final Cell cell : cells) {
> >             final long c = Bytes.toLong(
> >                     *cell.getQualifierArray()*,
> cell.getQualifierOffset(),
> > cell.getQualifierLength());
> >             final long v = Bytes.toLong(cell.getValueArray(),
> > cell.getValueOffset());
> >             points.put(c, v);
> >         }
> >     }
> > }
> >
> > The cell.getQualifierArray() method says that it's 'Max length is
> > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
> > 4,000 columnQualfiers.
> >
> > Is there an alternate API that I should be using or am I missing some
> > setting here? Note that in worst case I need to read all the
> > columnQualifiers in a row and I may or may not know a subset to fetch in
> > advance.
> >
> > Even if this is not possible in a single call, is there a way to cursor
> > through the columnQualifiers?
> >
> > I am presently using Hbase 0.96 client but can switch to Hbase 1.x if
> there
> > is an API in the newer version.
> >
> > --cheers, gaurav
> >
> > --
> > --cheers, gaurav
> >
>

--047d7bb04db076830005206e12b1--