Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 53C901041D for ; Fri, 20 Dec 2013 06:19:19 +0000 (UTC) Received: (qmail 14792 invoked by uid 500); 20 Dec 2013 06:19:16 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 14717 invoked by uid 500); 20 Dec 2013 06:19:16 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 14709 invoked by uid 99); 20 Dec 2013 06:19:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Dec 2013 06:19:16 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of xiaotao.cs.nju@gmail.com designates 209.85.216.170 as permitted sender) Received: from [209.85.216.170] (HELO mail-qc0-f170.google.com) (209.85.216.170) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Dec 2013 06:19:11 +0000 Received: by mail-qc0-f170.google.com with SMTP id x13so1907865qcv.1 for ; Thu, 19 Dec 2013 22:18:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=48f8b352AGRvVB8wUaTEqZ7xWE04Xpk4SV1Zw4s6IaU=; b=0vduMmqfZ3dM5PTmDcDJ2XiEpVka3myn3HqbKs9/nBFUZydfrO88vo/EZ7jcCFR/UU tXN639qOtPbo9j76tSIar/kPkjBh6nbmMQT4LipOfPZExQ8CDk3ienfl64WlGriFc10O 5VDBov7miNS+9O8EtrhEjSw0cH9BZoqsCqXGW/ZgNJ+ICgriw4iEoeVWPmyqZhvflFh8 GYvNTbIbSVCGH8/pWUExfgqo1+tZl9c9tUBjSqZjY1uoHJ3jUBRh0Aks8Ba+pQNzpDTt YbAXVd9YfNCxn+XDah5fh7XXE6arHnsfXxWTFc4sruu/ypJftpv1Jno5DbMGJ4SMYvUZ c9jQ== MIME-Version: 1.0 X-Received: by 10.49.110.202 with SMTP id ic10mr10792763qeb.18.1387520330739; Thu, 19 Dec 2013 22:18:50 -0800 (PST) Received: by 10.96.198.97 with HTTP; Thu, 19 Dec 2013 22:18:50 -0800 (PST) In-Reply-To: References: <12FBA326CCB5D446B61A2DDDCB41E42055B6C26E@szxema506-mbx.china.huawei.com> <12FBA326CCB5D446B61A2DDDCB41E42055B6C716@szxema506-mbx.china.huawei.com> <12FBA326CCB5D446B61A2DDDCB41E42055B6C8A0@szxema506-mbx.china.huawei.com> <12FBA326CCB5D446B61A2DDDCB41E42055B6CCFC@szxema506-mbx.china.huawei.com> Date: Fri, 20 Dec 2013 14:18:50 +0800 Message-ID: Subject: Re: Why so many unexpected files like partitions_xxxx are created? From: Tao Xiao To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=047d7bd76dfa6eb45404edf1401b X-Virus-Checked: Checked by ClamAV on apache.org --047d7bd76dfa6eb45404edf1401b Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable My HBase version is 0.94.12 2013/12/20 Tao Xiao > Hi Ted=A3=AC > You let me check the log of LoadIncrementalHFiles to see what was th= e > error from region server, but where is the log of LoadIncrementalHFiles? = Is > it in written into the log of region server? It seems the region server > works well > > > > > 2013/12/19 Ted Yu > >> From the stack trace posted I saw: >> >> org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:257) >> at >> >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionL= oad( >> LoadIncrementalHFiles.java:577) >> >> Assuming 0.94 is used, line 577 at the tip of 0.94 is: >> LOG.warn("Attempt to bulk load region containing " >> + Bytes.toStringBinary(first) + " into table " >> >> But the following should be the corresponding line w.r.t. stack trace: >> } catch (IOException e) { >> LOG.error("Encountered unrecoverable error from region server", e)= ; >> >> Tao: >> Can you check the log of LoadIncrementalHFiles to see what was the error >> from region server ? >> >> As Jieshan said, checking region server log would reveal something. >> >> Cheers >> >> >> On Tue, Dec 17, 2013 at 10:40 PM, Bijieshan wrote= : >> >> > It seems LoadIncrementalHFiles is still running. Can you run "jstack" >> on >> > 1 RegionServer process also? >> > >> > Which version are you using? >> > >> > Jieshan. >> > -----Original Message----- >> > From: Tao Xiao [mailto:xiaotao.cs.nju@gmail.com] >> > Sent: Wednesday, December 18, 2013 1:49 PM >> > To: user@hbase.apache.org >> > Subject: Re: Why so many unexpected files like partitions_xxxx are >> created? >> > >> > I did jstack one such process and can see the following output in the >> > terminal, and I guess this info told us that the processes started by >> the >> > command "LoadIncrementalHFiles" never exit. Why didn't they exit after >> > finished running ? >> > >> > ... ... >> > ... ... >> > >> > "LoadIncrementalHFiles-0.LruBlockCache.EvictionThread" daemon prio=3D1= 0 >> > tid=3D0x000000004129c000 nid=3D0x2186 in Object.wait() [0x00007f53f366= 5000] >> > java.lang.Thread.State: WAITING (on object monitor) >> > at java.lang.Object.wait(Native Method) >> > - waiting on <0x000000075fcf3370> (a >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread) >> > at java.lang.Object.wait(Object.java:485) >> > at >> > >> > >> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo= ckCache.java:631) >> > - locked <0x000000075fcf3370> (a >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread) >> > at java.lang.Thread.run(Thread.java:662) >> > >> > Locked ownable synchronizers: >> > - None >> > >> > "LoadIncrementalHFiles-3" prio=3D10 tid=3D0x00007f540ca55800 nid=3D0x2= 185 >> > runnable [0x00007f53f3765000] >> > java.lang.Thread.State: RUNNABLE >> > at java.io.FileOutputStream.writeBytes(Native Method) >> > at java.io.FileOutputStream.write(FileOutputStream.java:282) >> > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:10= 5) >> > - locked <0x0000000763e5af70> (a java.io.BufferedOutputStream) >> > at java.io.PrintStream.write(PrintStream.java:430) >> > - locked <0x0000000763d5b670> (a java.io.PrintStream) >> > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) >> > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263) >> > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106) >> > - locked <0x0000000763d6c6d0> (a java.io.OutputStreamWriter) >> > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:116) >> > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:203) >> > at java.io.Writer.write(Writer.java:140) >> > at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48) >> > at >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:317) >> > at org.apache.log4j.WriterAppender.append(WriterAppender.java:162) >> > at >> > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) >> > - locked <0x0000000763d5fb90> (a org.apache.log4j.ConsoleAppender) >> > at >> > >> > >> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(Ap= penderAttachableImpl.java:66) >> > at org.apache.log4j.Category.callAppenders(Category.java:206) >> > - locked <0x0000000763d65fe8> (a org.apache.log4j.spi.RootLogger) >> > at org.apache.log4j.Category.forcedLog(Category.java:391) >> > at org.apache.log4j.Category.log(Category.java:856) >> > at >> > org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:257= ) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionL= oad(LoadIncrementalHFiles.java:577) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncre= mentalHFiles.java:316) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncre= mentalHFiles.java:314) >> > at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto= r.java:895) >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:918) >> > at java.lang.Thread.run(Thread.java:662) >> > >> > Locked ownable synchronizers: >> > - <0x000000075fe494c0> (a >> > java.util.concurrent.locks.ReentrantLock$NonfairSync) >> > >> > ... ... >> > ... ... >> > >> > "Reference Handler" daemon prio=3D10 tid=3D0x00007f540c138800 nid=3D0x= 2172 in >> > Object.wait() [0x00007f5401355000] >> > java.lang.Thread.State: WAITING (on object monitor) >> > at java.lang.Object.wait(Native Method) >> > - waiting on <0x0000000763d51078> (a java.lang.ref.Reference$Lock) >> > at java.lang.Object.wait(Object.java:485) >> > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116= ) >> > - locked <0x0000000763d51078> (a java.lang.ref.Reference$Lock) >> > >> > Locked ownable synchronizers: >> > - None >> > >> > "main" prio=3D10 tid=3D0x00007f540c00e000 nid=3D0x216a waiting on cond= ition >> > [0x00007f54114ac000] >> > java.lang.Thread.State: WAITING (parking) >> > at sun.misc.Unsafe.park(Native Method) >> > - parking to wait for <0x000000075ea67310> (a >> > java.util.concurrent.FutureTask$Sync) >> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:15= 6) >> > at >> > >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterr= upt(AbstractQueuedSynchronizer.java:811) >> > at >> > >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInt= erruptibly(AbstractQueuedSynchronizer.java:969) >> > at >> > >> > >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInter= ruptibly(AbstractQueuedSynchronizer.java:1281) >> > at >> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218) >> > at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(Lo= adIncrementalHFiles.java:326) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadI= ncrementalHFiles.java:261) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncremen= talHFiles.java:780) >> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> > at >> > >> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncreme= ntalHFiles.java:785) >> > >> > Locked ownable synchronizers: >> > - None >> > >> > "VM Thread" prio=3D10 tid=3D0x00007f540c132000 nid=3D0x2170 runnable >> > >> > "Gang worker#0 (Parallel GC Threads)" prio=3D10 tid=3D0x00007f540c01c8= 00 >> > nid=3D0x216b runnable >> > >> > "Gang worker#1 (Parallel GC Threads)" prio=3D10 tid=3D0x00007f540c01e8= 00 >> > nid=3D0x216c runnable >> > >> > "Gang worker#2 (Parallel GC Threads)" prio=3D10 tid=3D0x00007f540c0200= 00 >> > nid=3D0x216d runnable >> > >> > "Gang worker#3 (Parallel GC Threads)" prio=3D10 tid=3D0x00007f540c0220= 00 >> > nid=3D0x216e runnable >> > >> > "Concurrent Mark-Sweep GC Thread" prio=3D10 tid=3D0x00007f540c0b1000 >> > nid=3D0x216f runnable "VM Periodic Task Thread" prio=3D10 >> > tid=3D0x00007f540c16b000 nid=3D0x217a waiting on condition >> > >> > JNI global references: 1118 >> > >> > >> > 2013/12/18 Ted Yu >> > >> > > Tao: >> > > Can you jstack one such process next time you see them hanging ? >> > > >> > > Thanks >> > > >> > > >> > > On Tue, Dec 17, 2013 at 6:31 PM, Tao Xiao >> > > wrote: >> > > >> > > > BTW, I noticed another problem. I bulk load data into HBase every >> > > > five minutes, but I found that whenever the following command was >> > executed >> > > > hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles >> > > > HFiles-Dir MyTable >> > > > >> > > > there is a new process called "LoadIncrementalHFiles" >> > > > >> > > > I can see many processes called "LoadIncrementalHFiles" using the >> > > > command "jps" in the terminal=A3=AC why are these processes still = there >> > > > even after the command that bulk load HFiles into HBase has finish= ed >> > > > executing ? I have >> > > to >> > > > kill them myself. >> > > > >> > > > >> > > > 2013/12/17 Bijieshan >> > > > >> > > > > Yes, it should be cleaned up. But not included in current code i= n >> > > > > my understanding. >> > > > > >> > > > > Jieshan. >> > > > > -----Original Message----- >> > > > > From: Ted Yu [mailto:yuzhihong@gmail.com] >> > > > > Sent: Tuesday, December 17, 2013 10:55 AM >> > > > > To: user@hbase.apache.org >> > > > > Subject: Re: Why so many unexpected files like partitions_xxxx a= re >> > > > created? >> > > > > >> > > > > Should bulk load task clean up partitions_xxxx upon completion ? >> > > > > >> > > > > Cheers >> > > > > >> > > > > >> > > > > On Mon, Dec 16, 2013 at 6:53 PM, Bijieshan >> > > wrote: >> > > > > >> > > > > > > I think I should delete these files immediately after I hav= e >> > > > > > > finished >> > > > > > bulk loading data into HBase since they are useless at that >> > > > > > time, >> > > > right ? >> > > > > > >> > > > > > Ya. I think so. They are useless once bulk load task finished. >> > > > > > >> > > > > > Jieshan. >> > > > > > -----Original Message----- >> > > > > > From: Tao Xiao [mailto:xiaotao.cs.nju@gmail.com] >> > > > > > Sent: Tuesday, December 17, 2013 9:34 AM >> > > > > > To: user@hbase.apache.org >> > > > > > Subject: Re: Why so many unexpected files like partitions_xxxx >> > > > > > are >> > > > > created? >> > > > > > >> > > > > > Indeed these files are produced by >> > org.apache.hadoop.hbase.mapreduce. >> > > > > > LoadIncrementalHFiles in the directory specified by what >> > > > > > job.getWorkingDirectory() >> > > > > > returns, and I think I should delete these files immediately >> > > > > > after I have finished bulk loading data into HBase since they >> > > > > > are useless at that time, right ? >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > 2013/12/16 Bijieshan >> > > > > > >> > > > > > > The reduce partition information is stored in this >> > > > > > > partition_XXXX >> > > > file. >> > > > > > > See the below code: >> > > > > > > >> > > > > > > HFileOutputFormat#configureIncrementalLoad: >> > > > > > > ..................... >> > > > > > > Path partitionsPath =3D new Path(job.getWorkingDirectory= (), >> > > > > > > "partitions_" + >> > > > UUID.randomUUID()); >> > > > > > > LOG.info("Writing partition information to " + >> > > > > > > partitionsPath); >> > > > > > > >> > > > > > > FileSystem fs =3D partitionsPath.getFileSystem(conf); >> > > > > > > writePartitions(conf, partitionsPath, startKeys); >> > > > > > > ..................... >> > > > > > > >> > > > > > > Hoping it helps. >> > > > > > > >> > > > > > > Jieshan >> > > > > > > -----Original Message----- >> > > > > > > From: Tao Xiao [mailto:xiaotao.cs.nju@gmail.com] >> > > > > > > Sent: Monday, December 16, 2013 6:48 PM >> > > > > > > To: user@hbase.apache.org >> > > > > > > Subject: Why so many unexpected files like partitions_xxxx a= re >> > > > created? >> > > > > > > >> > > > > > > I imported data into HBase in the fashion of bulk load, but >> > > > > > > after that I found many unexpected file were created in the >> > > > > > > HDFS >> > > directory >> > > > > > > of /user/root/, and they like these: >> > > > > > > >> > > > > > > /user/root/partitions_fd74866b-6588-468d-8463-474e202db070 >> > > > > > > /user/root/partitions_fd867cd2-d9c9-48f5-9eec-185b2e57788d >> > > > > > > /user/root/partitions_fda37b8a-a882-4787-babc-8310a969f85c >> > > > > > > /user/root/partitions_fdaca2f4-2792-41f6-b7e8-61a8a5677dea >> > > > > > > /user/root/partitions_fdd55baa-3a12-493e-8844-a23ae83209c5 >> > > > > > > /user/root/partitions_fdd85a3c-9abe-45d4-a0c6-76d2bed88ea5 >> > > > > > > /user/root/partitions_fe133460-5f3f-4c6a-9fff-ff6c62410cc1 >> > > > > > > /user/root/partitions_fe29a2b0-b281-465f-8d4a-6044822d960a >> > > > > > > /user/root/partitions_fe2fa6fa-9066-484c-bc91-ec412e48d008 >> > > > > > > /user/root/partitions_fe31667b-2d5a-452e-baf7-a81982fe954a >> > > > > > > /user/root/partitions_fe3a5542-bc4d-4137-9d5e-1a0c59f72ac3 >> > > > > > > /user/root/partitions_fe6a9407-c27b-4a67-bb50-e6b9fd172bc9 >> > > > > > > /user/root/partitions_fe6f9294-f970-473c-8659-c08292c27ddd >> > > > > > > ... ... >> > > > > > > ... ... >> > > > > > > >> > > > > > > >> > > > > > > It seems that they are HFiles, but I don't know why the were >> > > created >> > > > > > here? >> > > > > > > >> > > > > > > I bulk load data into HBase in the following way: >> > > > > > > >> > > > > > > Firstly, I wrote a MapReduce program which only has map >> tasks. >> > > The >> > > > > map >> > > > > > > tasks read some text data and emit them in the form of RowK= ey >> > > > > > > and KeyValue.The following is my program: >> > > > > > > >> > > > > > > @Override >> > > > > > > protected void map(NullWritable NULL, >> > > > > > > GtpcV1SignalWritable signal, Context ctx) throws >> > InterruptedException, IOException { >> > > > > > > String strRowkey =3D xxx; >> > > > > > > byte[] rowkeyBytes =3D Bytes.toBytes(strRowkey); >> > > > > > > >> > > > > > > rowkey.set(rowkeyBytes); >> > > > > > > >> > > > > > > part1.init(signal); >> > > > > > > part2.init(signal); >> > > > > > > >> > > > > > > KeyValue kv =3D new KeyValue(rowkeyBytes, Family= _A, >> > > > > > > Qualifier_Q, part1.serialize()); >> > > > > > > ctx.write(rowkey, kv); >> > > > > > > >> > > > > > > kv =3D new KeyValue(rowkeyBytes, Family_B, >> > > > > > > Qualifier_Q, part2.serialize()); >> > > > > > > ctx.write(rowkey, kv); >> > > > > > > } >> > > > > > > >> > > > > > > >> > > > > > > after the MR programs finished, there were several HFiles >> > > > > > > generated in the output directory I specified. >> > > > > > > >> > > > > > > Then I bean to load these HFiles into HBase using the >> > > > > > > following >> > > > > command: >> > > > > > > hbase >> > > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles >> > > > > > > HFiles-Dir MyTable >> > > > > > > >> > > > > > > Finally , I could see that the data were indeed loaded into >> > > > > > > the table in HBase. >> > > > > > > >> > > > > > > >> > > > > > > But, I could also see that there were many unexpected files >> > > > > > > generated in the HDFS directory of /user/root/, just as I >> > > > > > > have mentioned at the begining of this mail, and I did not >> > > > > > > specify any files to be produced in this directory. >> > > > > > > >> > > > > > > What happened ? Who can tell me what there files are and who >> > > > > > > produced >> > > > > > them? >> > > > > > > >> > > > > > > Thanks >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > > --047d7bd76dfa6eb45404edf1401b--