Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 66F0810F8B for ; Tue, 18 Nov 2014 23:09:35 +0000 (UTC) Received: (qmail 30577 invoked by uid 500); 18 Nov 2014 23:09:34 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 30469 invoked by uid 500); 18 Nov 2014 23:09:34 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 30164 invoked by uid 99); 18 Nov 2014 23:09:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Nov 2014 23:09:34 +0000 Date: Tue, 18 Nov 2014 23:09:34 +0000 (UTC) From: "Andrew Purtell (JIRA)" To: dev@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Reopened] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-10378?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-10378: ------------------------------------ Pardon if I'm implicating the wrong issue. Working on a Macbook dev box. I have found that some TestHRegion unit tests fail now because the can't ge= t a WAL writer: {noformat} $ mvn -DskipTests clean install $ mvn test -Dtest=3DTestHRegion Tests in error:=20 TestHRegion.testDelete_multiDeleteColumn:1836->initHRegion:4883->initHReg= ion:4904->initHRegion:4923 =C2=BB IOException cannot get log writer TestHRegion.testRegionReplicaSecondary:4587 =C2=BB IOException cannot get= log writer TestHRegion.testScanner_DeleteOneFamilyNotAnother:2096->initHRegion:4883-= >initHRegion:4904->initHRegion:4923 =C2=BB IOException cannot get log write= r TestHRegion.testScanner_JoinedScannersWithLimits:3156->initHRegion:4883->= initHRegion:4904->initHRegion:4923 =C2=BB IOException cannot get log writer {noformat} For example: {noformat} testRegionReplicaSecondary(org.apache.hadoop.hbase.regionserver.TestHRegion= ) Time elapsed: 0.063 sec <<< ERROR! java.io.IOException: cannot get log writer =09at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) =09at org.apache.hadoop.util.Shell.run(Shell.java:455) =09at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:= 702) =09at org.apache.hadoop.util.Shell.execCommand(Shell.java:791) =09at org.apache.hadoop.util.Shell.execCommand(Shell.java:774) =09at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys= tem.java:646) =09at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.jav= a:293) =09at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.jav= a:451) =09at org.apache.hadoop.fs.ChecksumFileSystem.createNonRecursive(ChecksumFi= leSystem.java:469) =09at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:10= 86) =09at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(Proto= bufLogWriter.java:90) =09at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWA= LProvider.java:361) =09at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(= FSHLog.java:736) =09at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.jav= a:701) =09at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.jav= a:612) =09at org.apache.hadoop.hbase.regionserver.wal.FSHLog.(FSHLog.java:54= 9) =09at org.apache.hadoop.hbase.wal.DefaultWALProvider.init(DefaultWALProvide= r.java:97) =09at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:13= 9) =09at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:171) =09at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.ja= va:4566) =09at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.ja= va:4533) =09at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.ja= va:4508) =09at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.ja= va:4585) =09at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.ja= va:4469) =09at org.apache.hadoop.hbase.regionserver.TestHRegion.testRegionReplicaSec= ondary(TestHRegion.java:4587) {noformat} Then, I can't clean: {noformat} $ mvn clean ... [ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:= 2.5:clean (default-clean) on project hbase-server: Failed to clean project:= Failed to delete /Users/apurtell/src/hbase/hbase-server/target/test-data/c= 8c12cda-613a-488d-9a9c-2463c8f884d5/WALs -> [Help 1] {noformat} I need to sudo rm -rf .=20 It wasn't like this last week. > Divide HLog interface into User and Implementor specific interfaces > ------------------------------------------------------------------- > > Key: HBASE-10378 > URL: https://issues.apache.org/jira/browse/HBASE-10378 > Project: HBase > Issue Type: Sub-task > Components: wal > Reporter: Himanshu Vashishtha > Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: 10378-1.patch, 10378-2.patch, HBASE-10378.3.patch.tx= t, HBASE-10378.4.patch.txt, HBASE-10378.5.patch.txt > > > HBASE-5937 introduces the HLog interface as a first step to support multi= ple WAL implementations. This interface is a good start, but has some limit= ations/drawbacks in its current state, such as: > 1) There is no clear distinction b/w User and Implementor APIs, and it pr= ovides APIs both for WAL users (append, sync, etc) and also WAL implementor= s (Reader/Writer interfaces, etc). There are APIs which are very much imple= mentation specific (getFileNum, etc) and a user such as a RegionServer shou= ldn't know about it. > 2) There are about 14 methods in FSHLog which are not present in HLog int= erface but are used at several places in the unit test code. These tests ty= pecast HLog to FSHLog, which makes it very difficult to test multiple WAL i= mplementations without doing some ugly checks. > I'd like to propose some changes in HLog interface that would ease the mu= lti WAL story: > 1) Have two interfaces WAL and WALService. WAL provides APIs for implemen= tors. WALService provides APIs for users (such as RegionServer). > 2) A skeleton implementation of the above two interface as the base class= for other WAL implementations (AbstractWAL). It provides required fields f= or all subclasses (fs, conf, log dir, etc). Make a minimal set of test only= methods and add this set in AbstractWAL. > 3) HLogFactory returns a WALService reference when creating a WAL instanc= e; if a user need to access impl specific APIs (there are unit tests which = get WAL from a HRegionServer and then call impl specific APIs), use Abstrac= tWAL type casting, > 4) Make TestHLog abstract and let all implementors provide their respecti= ve test class which extends TestHLog (TestFSHLog, for example). -- This message was sent by Atlassian JIRA (v6.3.4#6332)