Return-Path: Delivered-To: apmail-hbase-user-archive@www.apache.org Received: (qmail 18545 invoked from network); 10 Sep 2010 20:43:09 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 10 Sep 2010 20:43:09 -0000 Received: (qmail 1802 invoked by uid 500); 10 Sep 2010 20:43:08 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 1756 invoked by uid 500); 10 Sep 2010 20:43:08 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 1748 invoked by uid 99); 10 Sep 2010 20:43:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Sep 2010 20:43:08 +0000 X-ASF-Spam-Status: No, hits=0.6 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jgray@facebook.com designates 69.63.184.110 as permitted sender) Received: from [69.63.184.110] (HELO mx-out.facebook.com) (69.63.184.110) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Sep 2010 20:43:03 +0000 Received: from [10.18.255.129] ([10.18.255.129:20012] helo=mail.thefacebook.com) by mta006.ash1.facebook.com (envelope-from ) (ecelerity 2.2.2.45 r(34067)) with ESMTP id A3/5E-03153-2489A8C4; Fri, 10 Sep 2010 13:42:42 -0700 Received: from SC-MBX04.TheFacebook.com ([169.254.3.109]) by sc-hub04.TheFacebook.com ([fe80::8df5:7f90:d4a0:bb9%11]) with mapi; Fri, 10 Sep 2010 13:42:41 -0700 From: Jonathan Gray To: "user@hbase.apache.org" Subject: RE: Problem with bulk incremental loads.. Thread-Topic: Problem with bulk incremental loads.. Thread-Index: ActRF//s10h2RqXtTEufw+VhSUM8KgAEI/4Q Date: Fri, 10 Sep 2010 20:42:36 +0000 Message-ID: <5A76F6CE309AD049AAF9A039A3924282073DD880@sc-mbx04.TheFacebook.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 I ran into something like this as well but were in a rush to get the import= done so didn't look into it. I forgot about it so didn't follow up. We ended up ensuring regions would not be split during the job (configuring= the split size way up) and reran the MR job. JG > -----Original Message----- > From: Vidhyashankar Venkataraman [mailto:vidhyash@yahoo-inc.com] > Sent: Friday, September 10, 2010 11:43 AM > To: user@hbase.apache.org; hbase-user@hadoop.apache.org > Subject: Problem with bulk incremental loads.. >=20 > I was trying to bulk increment some files into a HBAse (0.89) table and > found this problem.. >=20 > If a file does not fit into any of the regions in the existing table, > then the tool gets into an infinite loop of splitting the files.. I > have attached a sample output.. Todd, is this a known issue? >=20 > Vidhya >=20 > 10/09/07 01:57:29 INFO mapreduce.LoadIncrementalHFiles: Trying to load > hfile=3Dhdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigCo > lumn/8375781572558986795 first=3D0000003511885973 last=3D0000003511999994 > 10/09/07 01:57:29 INFO mapreduce.LoadIncrementalHFiles: HFile at > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/8 > 375781572558986795 no longer fits inside a single region. Splitting... > 10/09/07 01:57:37 INFO mapreduce.LoadIncrementalHFiles: Successfully > split into new HFiles > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/fae6ef95297635e32e24c572bec9056e.bottom and > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/fae6ef95297635e32e24c572bec9056e.top > 10/09/07 01:57:37 INFO mapreduce.LoadIncrementalHFiles: Trying to load > hfile=3Dhdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigCo > lumn/_tmp/fae6ef95297635e32e24c572bec9056e.top first=3D0000003511885973 > last=3D0000003511999994 > 10/09/07 01:57:37 INFO mapreduce.LoadIncrementalHFiles: HFile at > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/fae6ef95297635e32e24c572bec9056e.top no longer fits inside a single > region. Splitting... > 10/09/07 01:57:44 INFO mapreduce.LoadIncrementalHFiles: Successfully > split into new HFiles > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/fae6ef95297635e32e24c572bec9056e.bottom and > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top > 10/09/07 01:57:44 INFO mapreduce.LoadIncrementalHFiles: Trying to load > hfile=3Dhdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigCo > lumn/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top > first=3D0000003511885973 last=3D0000003511999994 > 10/09/07 01:57:44 INFO mapreduce.LoadIncrementalHFiles: HFile at > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top no longer fits inside a > single region. Splitting... > 10/09/07 01:57:51 INFO mapreduce.LoadIncrementalHFiles: Successfully > split into new HFiles > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.bottom and > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top > 10/09/07 01:57:51 INFO mapreduce.LoadIncrementalHFiles: Trying to load > hfile=3Dhdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigCo > lumn/_tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top > first=3D0000003511885973 last=3D0000003511999994 > 10/09/07 01:57:51 INFO mapreduce.LoadIncrementalHFiles: HFile at > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top no longer fits > inside a single region. Splitting... > 10/09/07 01:57:59 INFO mapreduce.LoadIncrementalHFiles: Successfully > split into new HFiles > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.bottom and > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top > 10/09/07 01:57:59 INFO mapreduce.LoadIncrementalHFiles: Trying to load > hfile=3Dhdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigCo > lumn/_tmp/_tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top > first=3D0000003511885973 last=3D0000003511999994 > 10/09/07 01:57:59 INFO mapreduce.LoadIncrementalHFiles: HFile at > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top no longer fits > inside a single region. Splitting... > 10/09/07 01:58:06 INFO mapreduce.LoadIncrementalHFiles: Successfully > split into new HFiles > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.bottom and > hdfs://b3130080.yst.yahoo.net:4600/user/crawler/docd_inc_v1/bigColumn/_ > tmp/_tmp/_tmp/_tmp/_tmp/fae6ef95297635e32e24c572bec9056e.top