Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of fnord999@googlemail.com
 designates 209.85.216.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=IC88XCqgc+YbnHY0ydWeB9asJLgioVVgGwqAeL/11UUkId2SBippg9mCuhhmMkvYq7
         pUgN/1KtBzsOJ/5LSauNYX3KjJMvphzSw3YYEdhTYL+JMITwg7ACxseFQp1XdVHl5W8q
         dWDvNiLYn8w/jVILww+0SsZOpfXPsOCfumqfM=
MIME-Version: 1.0
Date: Mon, 22 Nov 2010 11:01:40 +0100
Message-ID: <AANLkTikoPsLatnSDaHW28ctExwCSxK7YqaO_nYP=WmL7@mail.gmail.com>
Subject: LazyFetching of Row Results in MapReduce
From: fnord 99 <fnord999@googlemail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf300fb4c1b694ac0495a157fc

--20cf300fb4c1b694ac0495a157fc
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I recently filled an hbase table with many millions of columns in each row
(!). The problem that now occured was that I always get a Heap Space Error
from the JVM with a subsequent shutdown of all regionservers in which the
error occurs. Since the error isn't thrown in any of my own classes, I think
that the problem is the following:

* a row is always completely read into memory upon access (at least all
column families that I'm interested in)
* the Result object holds the complete family-qualifier-value pairs in a
KeyValue[]
* this is sometimes too much to be handled by the physical memory each map
can get, therefore a heap space error is thrown

My question is now: is there any lazy fetching technique implemented within
the single key-values within one row? In my opinion it should be but I
couldn't find anything in the source code or wiki that hints to that.

Any ideas on how to go around this problem? I had the idea to rebuild the
table schema to store more data in the row key and less data in the column
families which would make the tables "thinner" and "longer". It would work
in the current setup, however, it wouldn't solve the original problem...

Thanks already in advance for any input on that,

fnord999

--20cf300fb4c1b694ac0495a157fc--