hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeetendra Gangele <gangele...@gmail.com>
Subject Re: Large number of column qualifiers
Date Wed, 23 Sep 2015 17:47:16 GMT
Hi Gaurav
I remember there is setting in abase-site.xml to increase this can you
please check it once or try to search?

On 23 September 2015 at 23:03, Gaurav Agarwal <gaurav@arkin.net> wrote:

> Hi Ted,
>
> Thanks for the reply. As I understood from 11544 description, it deals with
> optimising the server response by chunking it - in order to avoid very
> large memory allocations..
>
> In my case, I have enough memory to hold the results; that is not the
> problem. Rather the Cell object itself has a hard-limit of holding 32KB of
> columnQualifier metadata. In my case the number of columns (and thus their
> size) far exceeds this limit.
>
> Please let me know if I missed something in your suggestion.
>
> On Wed, Sep 23, 2015 at 10:54 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Please take a look at HBASE-11544 which is in hbase 1.1
> >
> > Cheers
> >
> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal <gaurav@arkin.net>
> wrote:
> >
> > > Hi All,
> > >
> > > I have Column Family with very large number of column qualifiers (>
> > > 50,000). Each column qualifier is 8 bytes long. The problem is the
> when I
> > > do a scan operation to fetch some rows, the client side Cell object
> does
> > > not have enough space allocated in it to hold all the columnQaulifiers
> > for
> > > a given row and hence I cannot read all the columns back for a given
> row.
> > >
> > > Please see the code snippet that I am using:
> > >
> > >  final ResultScanner rs = htable.getScanner(scan);
> > >  for (Result row = rs.next(); row != null; row = rs.next()) {
> > >     final Cell[] cells = row.rawCells();
> > >     if (cells != null) {
> > >         for (final Cell cell : cells) {
> > >             final long c = Bytes.toLong(
> > >                     *cell.getQualifierArray()*,
> > cell.getQualifierOffset(),
> > > cell.getQualifierLength());
> > >             final long v = Bytes.toLong(cell.getValueArray(),
> > > cell.getValueOffset());
> > >             points.put(c, v);
> > >         }
> > >     }
> > > }
> > >
> > > The cell.getQualifierArray() method says that it's 'Max length is
> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
> > > 4,000 columnQualfiers.
> > >
> > > Is there an alternate API that I should be using or am I missing some
> > > setting here? Note that in worst case I need to read all the
> > > columnQualifiers in a row and I may or may not know a subset to fetch
> in
> > > advance.
> > >
> > > Even if this is not possible in a single call, is there a way to cursor
> > > through the columnQualifiers?
> > >
> > > I am presently using Hbase 0.96 client but can switch to Hbase 1.x if
> > there
> > > is an API in the newer version.
> > >
> > > --cheers, gaurav
> > >
> > > --
> > > --cheers, gaurav
> > >
> >
>
>
>
> --
> --cheers, gaurav
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message