hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric <eric.x...@gmail.com>
Subject Re: KeyValue size too large
Date Mon, 17 Jan 2011 12:38:04 GMT
St.Ack, my files ranged from 1KB up to 80MB sometimes. Maybe the cluster was
also underpowered (3 nodes). I'm now looking at writing the files to map
files and only storing meta information.

2011/1/15 Stack <stack@duboce.net>

> Tell us more Eric.  Which HBase version?  What kinda sizes.  We should
> make it work reliably for at least cells of 1 or 2 MB.
> Thanks,
> St.Ack
>
> On Sat, Jan 15, 2011 at 2:55 AM, Eric <eric.xkcd@gmail.com> wrote:
> > I would also recommend against storing large files in HBase. Your regions
> > get filled up very quickly, meaning you will get a lot of regions. You
> can
> > increase the max region size but I've seen very unstable behaviour while
> > insering tens of thousands of small to large files into HBase with both
> > small and large max region sizes.
> >
> > 2011/1/15 Ryan Rawson <ryanobjc@gmail.com>
> >
> >> This is a client side check yes, requested as a double check to make
> >> sure that applications aren't going crazy.
> >>
> >> Its a combination size of row, qualifier, HBase metadata, and your
> >> actual value.  If the combo exceeds that limit you get that exception.
> >>
> >> Disable it by setting it to 0.  Note that disabling this check doesn't
> >> actually lift any intrinsic limits that are due to use integer and
> >> other fields to store lengths.  That is, HBase is not appropriate for
> >> storing cells > 2GB, and you will hit performance problems before then
> >> anyways.
> >>
> >> For extremely large data, consider storing the meta data in HBase and
> >> the actual data in HDFS.
> >>
> >> -ryan
> >>
> >> On Fri, Jan 14, 2011 at 4:09 PM, Sean Bigdatafun
> >> <sean.bigdatafun@gmail.com> wrote:
> >> > And I guess if your k-v exceed that number. The HBase client library
> will
> >> > cut at the the last byte on the boundary? (my question is if this
> >> parameter
> >> > is a client side parameter)
> >> >
> >> >
> >> >
> >> > On Fri, Jan 14, 2011 at 11:15 AM, Christopher Tarnas <cft@email.com>
> >> wrote:
> >> >
> >> >> You need to increase the hbase.client.keyvalue.maxsize property in
> >> >> hbase-site.xml. It defaults to 10 MB.
> >> >>
> >> >> -chris
> >> >>
> >> >> On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
> >> >> <srini@qualcomm.com>wrote:
> >> >>
> >> >> > Hi,
> >> >> > Running a throughput test with various sample sizes but getting
> >> "KeyValue
> >> >> > size too large" for 10MB files. Could not remember any setting
> related
> >> to
> >> >> > cell max size.
> >> >> > Thanks
> >> >> > + Srini  Sadasivam
> >> >> >
> >> >> > 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0
req/s
> >> Total
> >> >> > Througput: 0 req/s
> >> >> > Exception in thread "Thread-10" java.lang.IllegalArgumentException:
> >> >> > KeyValue size too large
> >> >> >        at
> >> >> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
> >> >> >        at
> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
> >> >> >        at
> org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
> >> >> >        at
> >> >> >
> >> >>
> >>
> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > --Sean
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message