Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9EEE11095F for ; Wed, 18 Dec 2013 02:59:26 +0000 (UTC) Received: (qmail 12837 invoked by uid 500); 18 Dec 2013 02:59:24 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 12724 invoked by uid 500); 18 Dec 2013 02:59:24 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 12712 invoked by uid 99); 18 Dec 2013 02:59:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Dec 2013 02:59:24 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [80.237.133.18] (HELO wp249.webpack.hosteurope.de) (80.237.133.18) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Dec 2013 02:59:06 +0000 Received: from c-24-7-17-25.hsd1.ca.comcast.net ([24.7.17.25] helo=[10.0.0.10]); authenticated by wp249.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) id 1Vt7LQ-0004Z5-TW; Wed, 18 Dec 2013 03:58:45 +0100 From: Timo Schaepe Content-Type: multipart/signed; boundary="Apple-Mail=_89E382AC-3969-49D6-877B-B6928FD81DCC"; protocol="application/pkcs7-signature"; micalg=sha1 Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: Problems with hbase.hregion.max.filesize Date: Tue, 17 Dec 2013 18:58:42 -0800 References: <2DE0F4DD-5892-4D84-9EFA-DE4F0893C6BE@timoschaepe.de> <1A950B54-BF53-4D73-8348-EC543525A44B@timoschaepe.de> <640C68E8-E23B-41AF-8B5E-F127D35A2ADA@timoschaepe.de> To: user@hbase.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1510) X-bounce-key: webpack.hosteurope.de;timo@timoschaepe.de;1387335546;8aa7c4bc; X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_89E382AC-3969-49D6-877B-B6928FD81DCC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hey Azuryy Yu, yep, checked the GC log, nothing there. I think, there is no special JVM configuration: export HBASE_OPTS=3D"-XX:+UseConcMarkSweepGC" export SERVER_GC_OPTS=3D"-verbose:gc -XX:+PrintGCDetails = -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation = -XX:NumberOfGCLogFiles=3D1 -XX:GCLogFileSize=3D512M = -Xloggc:/home/hadoop/logs/g c-hbase.log $HBASE_GC_OPTS" Thank, Timo Am 14.12.2013 um 15:45 schrieb Azuryy Yu : > such a large java heap, did you check gc log? how did you configured = jvm > options? > On 2013-12-14 9:27 PM, "Timo Schaepe" wrote: >=20 >> Sorry, forgot our Hardwareconfiguration=85 >>=20 >> 1 NameNode/SecondaryNameNode/HBase master >> 31 Datanodes/Regionserver >>=20 >> All of them with >> 2x XEON E5-2640 2.5 GHz >> 128 GB RAM >> /dev/sda 90 GB >> /dev/sdb 1.1 TB >> /dev/sdc 1.1 TB >>=20 >> where sda are SSDs disks for system and sdb and sdc are disks for >> HDFS/HBase >>=20 >> Heapsize for Regionserver: 80 GB >>=20 >> bye, >>=20 >> Timo >>=20 >>=20 >> Am 14.12.2013 um 14:21 schrieb Timo Schaepe : >>=20 >>> Hey, >>>=20 >>> @JM: Thanks for the hint with hbase.regionserver.fileSplitTimeout. = At >> the moment (the import is actually working) and after I splittet the >> specific regions manually, we do not have growing regions anymore. >>>=20 >>> hbase hbck says, all things are going fine. >>> 0 inconsistencies detected. >>> Status: OK >>>=20 >>> @Ted Yu: Sure, have a look here: http://pastebin.com/2ANFVZEU >>> The relevant tablename ist data_1091. >>>=20 >>> Thanks for your time. >>>=20 >>> Timo >>>=20 >>> Am 13.12.2013 um 20:18 schrieb Ted Yu : >>>=20 >>>> Timo: >>>> Can you pastebin regionserver log around 2013-12-12 13:54:20 so = that we >> can >>>> see what happened ? >>>>=20 >>>> Thanks >>>>=20 >>>>=20 >>>> On Fri, Dec 13, 2013 at 11:02 AM, Jean-Marc Spaggiari < >>>> jean-marc@spaggiari.org> wrote: >>>>=20 >>>>> Try to increase hbase.regionserver.fileSplitTimeout but put it = back to >> its >>>>> default value after. >>>>>=20 >>>>> Default value is 30 seconds. I think it's not normal for a split = to >> take >>>>> more than that. >>>>>=20 >>>>> What is your hardware configuration? >>>>>=20 >>>>> Have you run hbck to see if everything is correct? >>>>>=20 >>>>> JM >>>>>=20 >>>>>=20 >>>>> 2013/12/13 Timo Schaepe >>>>>=20 >>>>>> Hello again, >>>>>>=20 >>>>>> digging in the logs of the specific regionserver shows me that: >>>>>>=20 >>>>>> 2013-12-12 13:54:20,194 INFO >>>>>> org.apache.hadoop.hbase.regionserver.SplitRequest: Running >>>>> rollback/cleanup >>>>>> of failed split of >>>>>>=20 >>>>>=20 >> = data,OR\x83\xCF\x02\x82\xAE\xF3U,1386851456415.d04cadb1b2252dafc476c138e96= 51ca7.; >>>>>> Took too long to split the files and create the references, = aborting >>>>> split >>>>>>=20 >>>>>> This message appears two time, so it seems, that HBase tried to = split >> the >>>>>> region but it failed. I don't know why. How is the behaviour of = HBase, >>>>> if a >>>>>> region split fails? Are there more tries to split this region = again? I >>>>>> didn't find any new tries in the log. Now I split the big regions >>>>> manually >>>>>> and this works. And also it seems, that HBase split the new = regions >> again >>>>>> to crunch they down to the given limit. >>>>>>=20 >>>>>> But also it is a mystery for me, why the split size in Hannibal = shows >> me >>>>>> 10 GB and in base-site.xml I put 2 GB=85 >>>>>>=20 >>>>>> Thanks, >>>>>>=20 >>>>>> Timo >>>>>>=20 >>>>>>=20 >>>>>> Am 13.12.2013 um 10:22 schrieb Timo Schaepe = : >>>>>>=20 >>>>>>> Hello, >>>>>>>=20 >>>>>>> during the loading of data in our cluster I noticed some strange >>>>>> behavior of some regions, that I don't understand. >>>>>>>=20 >>>>>>> Scenario: >>>>>>> We convert data from a mysql database to HBase. The data is = inserted >>>>>> with a put to the specific HBase table. The row key is a = timestamp. I >>>>> know >>>>>> the problem with timestamp keys, but in our requirement it works = quiet >>>>>> well. The problem is now, that there are some regions, which are >> growing >>>>>> and growing. >>>>>>>=20 >>>>>>> For example the table on the picture [1]. First, all data was >>>>>> distributed over regions and node. And now, the data is written = into >> only >>>>>> one region, which is growing and I can see no splitting at all. >> Actually >>>>>> the size of the big region is nearly 60 GB. >>>>>>>=20 >>>>>>> HBase version is 0.94.11. I cannot understand, why the splitting = is >> not >>>>>> happening. In hbase-site.xml I limit the = hbase.hregion.max.filesize >> to 2 >>>>> GB >>>>>> and HBase accepted this value. >>>>>>>=20 >>>>>>> >>>>>>> >>>>>>> hbase.hregion.max.filesize >>>>>>> 2147483648 >>>>>>> >>>>>>>=20 >>>>>>> First mystery: Hannibal shows me the split size is 10 GB (see >>>>>> screenshot). >>>>>>> Second mystery: HBase is not splitting some regions neither at 2 = GB >> nor >>>>>> 10 GB. >>>>>>>=20 >>>>>>> Any ideas? Could be the timestamp rowkey cause this problem? >>>>>>>=20 >>>>>>> Thanks, >>>>>>>=20 >>>>>>> Timo >>>>>>>=20 >>>>>>> [1] https://www.dropbox.com/s/lm286xkcpglnj1t/big_region.png >>>>>>=20 >>>>>>=20 >>>>>=20 >>>=20 >>=20 >>=20 --Apple-Mail=_89E382AC-3969-49D6-877B-B6928FD81DCC Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMuDCCBVcw ggM/oAMCAQICAwEmOzANBgkqhkiG9w0BAQUFADBUMRQwEgYDVQQKEwtDQWNlcnQgSW5jLjEeMBwG A1UECxMVaHR0cDovL3d3dy5DQWNlcnQub3JnMRwwGgYDVQQDExNDQWNlcnQgQ2xhc3MgMyBSb290 MB4XDTEzMDYxMDEyMzc0MVoXDTE1MDYxMDEyMzc0MVowOjEUMBIGA1UEAxQLVGltbyBTY2jkcGUx IjAgBgkqhkiG9w0BCQEWE3RpbW9AdGltb3NjaGFlcGUuZGUwggEiMA0GCSqGSIb3DQEBAQUAA4IB DwAwggEKAoIBAQCVyMKfXb8DElCZxbijCX6djB6F14C1gOewDMbzBcQPmhmjyvLFzQq83KQEh2lZ /62r+0hVUvuedIYHRCDZSDzARfAajoDiiYZcpIq/pDv1tr4YOgvw6XBgcpd2T+BLMMqArxItj5zz fuU8rGLDQzE23v6pNd4NXwh7oXtg2ONsSH3Hzv8Qtnit8JeRwxwSyWbEuzs4QUNRxlVcXSfcoI22 o7bH1UWGR6uXiq1pu5OAVtREcE2ezuDJrpkfOKXDf61YAuo/vN7I98AFCjNCOCbgvjozJ3qtj5N6 EkLiGh3bfZ6w4OFsX5ppaOnEiZbIh5OeWFLQYQAKn/0JYQGhbRPdAgMBAAGjggFKMIIBRjAMBgNV HRMBAf8EAjAAMFYGCWCGSAGG+EIBDQRJFkdUbyBnZXQgeW91ciBvd24gY2VydGlmaWNhdGUgZm9y IEZSRUUgaGVhZCBvdmVyIHRvIGh0dHA6Ly93d3cuQ0FjZXJ0Lm9yZzAOBgNVHQ8BAf8EBAMCA6gw QAYDVR0lBDkwNwYIKwYBBQUHAwQGCCsGAQUFBwMCBgorBgEEAYI3CgMEBgorBgEEAYI3CgMDBglg hkgBhvhCBAEwMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzABhhZodHRwOi8vb2NzcC5jYWNlcnQu b3JnMDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuY2FjZXJ0Lm9yZy9jbGFzczMtcmV2b2tl LmNybDAeBgNVHREEFzAVgRN0aW1vQHRpbW9zY2hhZXBlLmRlMA0GCSqGSIb3DQEBBQUAA4ICAQAS MbSPnH66AzS4wd28RRtP38pjR8m50uE3TQtF0fI7VI31dDp6EgJIOhSRcQeDY+7FW0WGtuGmJ63S XOKoZEDMkiG/vqexYdzrfpWK0HuoouIrnvqpgKKHlc/bXU6G+WfykKEL3876l2R7g0VjEnEZClBg kJYKx82qg5CXZCdmpR17YQwSHTJu41t4rjNxNB8b8q0/e/TmVhNfgbWYqID5ZJJSlVhBAWqDJZ9x EUU1Ns8dVJ/OPQjHNP0KWiY1+UMoPkDFo3aQE9kSk0fcHhqlY++xgFZE6eExjmKYo0FXmiAWuhUv wCzS8hzfE58Z63cTGLiIL8aY3D0Dfgv2WbDPrjs9jwIHiJOYqnxmdkwZCyHpJGgUsumOHEtDIsMA DfSF4Q5IhIGmmR+WoEgRSLIH0V0NVFU1XSjfdIgPWXcKXHm8GWhFz710a985hAZMTngj4swCOBx4 yKp29W2mVnbmrfass3i9pnoxkDssFEpnpcDFrQP9PmJhJvV/DW/ZAGM3BjlUJ4k2AIhLfH/s/JLP fzF8ObGqwZNlUZ696qjwBnwo3TK83PtAIefQ79ma1hWnO91pao8SygA/sOeMW4auYeBa49Jv3ROW JB1Ed78/c/3PVNMlWp+p7qzagqtd+bhbPTKu2Q/sGgXHit72HBw9hrA0gsKjEFmY3BotNahWITCC B1kwggVBoAMCAQICAwpBijANBgkqhkiG9w0BAQsFADB5MRAwDgYDVQQKEwdSb290IENBMR4wHAYD VQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBBdXRo b3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0xMTA1MjMxNzQ4MDJa Fw0yMTA1MjAxNzQ4MDJaMFQxFDASBgNVBAoTC0NBY2VydCBJbmMuMR4wHAYDVQQLExVodHRwOi8v d3d3LkNBY2VydC5vcmcxHDAaBgNVBAMTE0NBY2VydCBDbGFzcyAzIFJvb3QwggIiMA0GCSqGSIb3 DQEBAQUAA4ICDwAwggIKAoICAQCrSTURSHzSJn5TlM9Dqd0o10Iqi/OHeBlYfA+e2ol94fvrcpAN dKGWZKufoCSZc9riVXbHF3v1BKxGuMO+f2SNEGwk82GcwPKQ+lHm9WkBY8MPVuJKQs/iRIwlKKjF eQl9RrmK8+nzNCkIReQcn8uUBByBqBSzmGXEQ+xOgo0J0b2qW42S0OzekMV/CsLj6+YxWl50Ppcz WejDAz1gM7/30W9HxM3uYoNSbi4ImqTZFRiRpoWSR7CuSOtttyHshRpocjWr//AQXcD0lKdq1TuS fkyQBX6TwSyLpI5idBVxbgtxA+qvFTia1NIFcm+M+SvrWnIl+TlG43IbPgTDZCciECqKT1inA62+ tC4T7V2qSNfVfdQqe1z6RgRQ5MwOQluM7dvyz/yWk+DbETZUYjQ4jwxgmzuXVjit89Jbi6Bb6k6W uHzX1aCGcEDTkSm3ojyt9Yy7zxqSiuQ0e8DYbF/pCsLDpyCaWt8sXVJcukfVm+8kKHA4IC/VfynA skEDaJLM4JzMl0tF7zoQCqtwOpiVcK01seqFK6QcgCExqa5geoAmSAC4AcCTY1UikTxW56/bOiXz jzFU6iaLgVn5odFTEcV7nQP2dBHgbbEsPyyGkZlxmqZ3izRg0RS0LKydr4wQ05/EavhvE/xzWfdm QnQeiuP43NJvmJzLR5iVQAX76QIDAQABo4ICDTCCAgkwHQYDVR0OBBYEFHWocWBMiBPweNmJd7Vt xYnfvLF6MIGjBgNVHSMEgZswgZiAFBa1MhvUx/Pg5o7zvdKwOu6yORjRoX2kezB5MRAwDgYDVQQK EwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENl cnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZ4IB ADAPBgNVHRMBAf8EBTADAQH/MF0GCCsGAQUFBwEBBFEwTzAjBggrBgEFBQcwAYYXaHR0cDovL29j c3AuQ0FjZXJ0Lm9yZy8wKAYIKwYBBQUHMAKGHGh0dHA6Ly93d3cuQ0FjZXJ0Lm9yZy9jYS5jcnQw SgYDVR0gBEMwQTA/BggrBgEEAYGQSjAzMDEGCCsGAQUFBwIBFiVodHRwOi8vd3d3LkNBY2VydC5v cmcvaW5kZXgucGhwP2lkPTEwMDQGCWCGSAGG+EIBCAQnFiVodHRwOi8vd3d3LkNBY2VydC5vcmcv aW5kZXgucGhwP2lkPTEwMFAGCWCGSAGG+EIBDQRDFkFUbyBnZXQgeW91ciBvd24gY2VydGlmaWNh dGUgZm9yIEZSRUUsIGdvIHRvIGh0dHA6Ly93d3cuQ0FjZXJ0Lm9yZzANBgkqhkiG9w0BAQsFAAOC AgEAKSiFrkSpua+keRPwqKMrl2DzXO7jL8H24magEa42Nzp2FQRT6kL1+erAFdimgtnkYa5yCylc kEPoQbLhd9sCE0R4R1WvWPzMmPZFudEg+NghB/5tqnPUs8YH6QmFzDvytr4sHCXVcYw5tS7qvhiB urCTuA/j5tcmjDFacgOEUuam9TMiRQrICw2KuDZvkAmhq73X1U4ucaLUrvqnVCvrNY1at1SIL+50 n+1IFsoNSNCU06ykovYk35LjvetDQJFuHBiOVrSCEvOpk5/UvJytnHXuWpcbled0LRwPsCyXn/up Mzl65wM6ko4i9owN5Nl+DXYY9wH575aWolVzwDxxtB0aVkO3wwqNcvziEAkLQc6MlKD5A/1xc0uK VzPljnR+FQEA5sxKHOd/lRktxaUMi7u17YWzXNPfuLnyyscNARSscFjFjI0z1J1moxpQlSP8SOAG QxLZzaeGOS82cqOAEOTh89HLWxrA5ICafBNzBk/bo2skCrqzHLxKeLvl43U4pUinoh6vdtRe9ziG VlqJztbDp3myUqDG8YW0JYzyP5azENmNbFc7n2+GOhiCIjbIsJE42yqhk6qEP/UnZa5z1cjV03fq S53HQbvHwOOgP+R9pI1z5hJL36Fzc3M6gOjVy44vy+oTp9ZBi6z6PInXJPVOtOBhkrfzN5jEvpaj t4oxggK9MIICuQIBATBbMFQxFDASBgNVBAoTC0NBY2VydCBJbmMuMR4wHAYDVQQLExVodHRwOi8v d3d3LkNBY2VydC5vcmcxHDAaBgNVBAMTE0NBY2VydCBDbGFzcyAzIFJvb3QCAwEmOzAJBgUrDgMC GgUAoIIBNzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xMzEyMTgw MjU4NDNaMCMGCSqGSIb3DQEJBDEWBBTxR+pNNRaZhd/UcyD50PHoWCS9BDBqBgkrBgEEAYI3EAQx XTBbMFQxFDASBgNVBAoTC0NBY2VydCBJbmMuMR4wHAYDVQQLExVodHRwOi8vd3d3LkNBY2VydC5v cmcxHDAaBgNVBAMTE0NBY2VydCBDbGFzcyAzIFJvb3QCAwEmOzBsBgsqhkiG9w0BCRACCzFdoFsw VDEUMBIGA1UEChMLQ0FjZXJ0IEluYy4xHjAcBgNVBAsTFWh0dHA6Ly93d3cuQ0FjZXJ0Lm9yZzEc MBoGA1UEAxMTQ0FjZXJ0IENsYXNzIDMgUm9vdAIDASY7MA0GCSqGSIb3DQEBAQUABIIBABt2epM4 WjeiwG7ucChrnLWHTAAX8WZm3iYAFmEJs3xTwEa84eg1Lnq0gMVXAN9K9mHje1tZaJT1Hd4QvKEr 83ysxw5y5lEgNo0JqZS9rYKaK+N4kSlMJiDMzGCUkOc2UMJFSlbHmYGFcuwHFwlFlsq3Mad9D0Yl sUqZo/hAspvgZvnL5w+2aBQpDt/Oq9+0GBB58lvg0vEk61l8QfHs06+n/CtH8s/o28oaWfxSffYc sjkjj/mdW7+Y5QXUlVypSH38qlWNq7Gr5UpqUewh+TT3/yP6wN6OIfBXxsWVqh9WhoxNYCcKRoEL cx3qZqNht6IU7e7tbIZiGGb4gBg97xUAAAAAAAA= --Apple-Mail=_89E382AC-3969-49D6-877B-B6928FD81DCC--