Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of vrodionov@carrieriq.com
 designates 204.235.122.16 as permitted sender)
From: Vladimir Rodionov <vrodionov@carrieriq.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Date: Wed, 28 May 2014 20:10:48 -0700
Subject: RE: about "hbase.client.keyvalue.maxsize"
Thread-Topic: about "hbase.client.keyvalue.maxsize"
Thread-Index: Ac965VUa9Xzj676MTx+2/LxEXRZbGgABkG8w
Message-ID: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15187CA57C69@exchange2007.carrieriq.com>
References: 
 <53DC189E5FAEFA43BFA1BC02431031DB740A3A1381@WECTMLBOX.winbond.com.tw>
In-Reply-To: 
 <53DC189E5FAEFA43BFA1BC02431031DB740A3A1381@WECTMLBOX.winbond.com.tw>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I suggest you reconsider your application design if you need to store large=
 objects. Keep these object outside HBase and store in HBase references and=
/or indexes.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: Henry Hung [YTHung1@winbond.com]
Sent: Wednesday, May 28, 2014 7:26 PM
To: user@hbase.apache.org
Subject: about "hbase.client.keyvalue.maxsize"

Hi All,

Today I stumble upon this error:

Error: java.io.IOException: java.io.IOException: java.lang.IllegalArgumentE=
xception: KeyValue size too large
        at com.winbond.hadoop.fdc.mapreduce.xml.XmlToHBaseMapper.map(XmlToH=
BaseMapper.java:204)
        at com.winbond.hadoop.fdc.mapreduce.xml.XmlToHBaseMapper.map(XmlToH=
BaseMapper.java:1)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupIn=
formation.java:1491)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

After looking into the source code, it appears this is a constraint impose =
by hbase.client.keyvalue.maxsize =3D 10485760.

>From the parameter description:
Specifies the combined maximum allowed size of a KeyValue instance. This is=
 to set an upper boundary for a single entry saved in a storage file. Since=
 they cannot be split it helps avoiding that a region cannot be split any f=
urther because the data is too large. It seems wise to set this to a fracti=
on of the maximum region size. Setting it to zero or less disables the chec=
k.

If I set the table's region size to be maximum 20GB before splitting, could=
 I set the hbase.client.keyvalue.maxsize to a larger value? Such as: 200 MB

One more thing, when I looked into HTable.java source code, it appears that=
 the key value size is checked before compression, is this true?
I think it should be more reasonable to check the size after compression, n=
o?

Best regards,
Henry Hung

________________________________
The privileged confidential information contained in this email is intended=
 for use only by the addressees as indicated by the original sender of this=
 email. If you are not the addressee indicated in this email or are not res=
ponsible for delivery of the email to such a person, please kindly reply to=
 the sender indicating this fact and delete all copies of it from your comp=
uter and network server immediately. Your cooperation is highly appreciated=
. It is advised that any unauthorized use of confidential information of Wi=
nbond is strictly prohibited; and any information in this email irrelevant =
to the official business of Winbond shall be deemed as neither given nor en=
dorsed by Winbond.

Confidentiality Notice:  The information contained in this message, includi=
ng any attachments hereto, may be confidential and is intended to be read o=
nly by the individual or entity to whom this message is addressed. If the r=
eader of this message is not the intended recipient or an agent or designee=
 of the intended recipient, please note that any review, use, disclosure or=
 distribution of this message or its attachments, in any form, is strictly =
prohibited.  If you have received this message in error, please immediately=
 notify the sender and/or Notifications@carrieriq.com and delete or destroy=
 any copy of this message and its attachments.