From issues-return-161132-archive-asf-public=cust-asf.ponee.io@hive.apache.org Mon Jun 24 11:24:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0AB30180671 for ; Mon, 24 Jun 2019 13:24:01 +0200 (CEST) Received: (qmail 22218 invoked by uid 500); 24 Jun 2019 11:24:01 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 22209 invoked by uid 99); 24 Jun 2019 11:24:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jun 2019 11:24:01 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B6309E2DF6 for ; Mon, 24 Jun 2019 11:24:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 51DF22544E for ; Mon, 24 Jun 2019 11:24:00 +0000 (UTC) Date: Mon, 24 Jun 2019 11:24:00 +0000 (UTC) From: "Ashutosh Bapat (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-21880) Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-21880?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21880: ---------------------------------- Attachment: HIVE-21880.01.patch Status: Patch Available (was: Open) The code in=C2=A0getNextNotification() just checks whether the next event h= as the expected event id. This check may fail when there are multiple event= s with the same event id or when event ids are missing.=C2=A0When the test = fails, it fails because there multiple events with the same event id. We use=C2=A0derby database as backing db for metastore. Derby doesn't lock = the row being selected with FOR UPDATE clause. addNotificationLog() and add= NotificationEvent(), both functions, rely on the this behaviour to generate= monotonically increasing sequential event ids. Since the row is not locked= , we could fetch the same event id multiple times and then increment it to = the same value multiple times. That can cause the event ids to progress in = unreliable manner. So for Derby we lock the NOTIFICATION_SEQUENCE table ins= tead of using FOR UPDATE. Note: TxnHandler uses a different behaviour to simulate the effect of FOR U= PDATE on Derby; it uses a JVM wide mutex for that.=C2=A0TxnHandler is not a= vailable always esp. when there are no ACID tables involved, so we need to = move that mutex out of TxnHandler to a place common to DbNotificationListen= er and TxnHandler e.g. SQLGenerater and also have to take care of mutex's r= eentrant behaviour. Furthermore such a mutex wouldn't work when there are m= etastores are running in separate JVMs. Since the test in Subject is flaky, I have added another test which reliabl= y reproduces this behaviour. > Enable flaky test TestReplicationScenariosAcidTablesBootstrap.testBootstr= apAcidTablesDuringIncrementalWithConcurrentWrites. > -------------------------------------------------------------------------= -------------------------------------------------- > > Key: HIVE-21880 > URL: https://issues.apache.org/jira/browse/HIVE-21880 > Project: Hive > Issue Type: Bug > Components: repl > Affects Versions: 4.0.0 > Reporter: Sankar Hariappan > Assignee: Ashutosh Bapat > Priority: Major > Labels: pull-request-available > Attachments: HIVE-21880.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Need tp enable TestReplicationScenariosAcidTablesBootstrap.testBootstrapA= cidTablesDuringIncrementalWithConcurrentWrites which is disabled as it is f= laky and randomly failing with below error. > {code} > Error Message > Notification events are missing in the meta store. > Stacktrace > java.lang.IllegalStateException: Notification events are missing in the m= eta store. > =09at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotific= ation(HiveMetaStoreClient.java:3246) > =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp= l.java:62) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:498) > =09at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(Ret= ryingMetaStoreClient.java:212) > =09at com.sun.proxy.$Proxy58.getNextNotification(Unknown Source) > =09at org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotifi= cationFetcher.getNextNotificationEvents(EventUtils.java:107) > =09at org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEv= entIterator.fetchNextBatch(EventUtils.java:159) > =09at org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEv= entIterator.hasNext(EventUtils.java:189) > =09at org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.incrementalDump(Re= plDumpTask.java:231) > =09at org.apache.hadoop.hive.ql.exec.repl.ReplDumpTask.execute(ReplDumpTa= sk.java:121) > =09at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) > =09at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.= java:103) > =09at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2709) > =09at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2361) > =09at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2028) > =09at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1788) > =09at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1782) > =09at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java= :162) > =09at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java= :223) > =09at org.apache.hadoop.hive.ql.parse.WarehouseInstance.run(WarehouseInst= ance.java:227) > =09at org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseIns= tance.java:282) > =09at org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseIns= tance.java:265) > =09at org.apache.hadoop.hive.ql.parse.WarehouseInstance.dump(WarehouseIns= tance.java:289) > =09at org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesB= ootstrap.testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites(TestR= eplicationScenariosAcidTablesBootstrap.java:328) > =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp= l.java:62) > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc= essorImpl.java:43) > =09at java.lang.reflect.Method.invoke(Method.java:498) > =09at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framewo= rkMethod.java:47) > =09at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveC= allable.java:12) > =09at org.junit.runners.model.FrameworkMethod.invokeExplosively(Framework= Method.java:44) > =09at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeM= ethod.java:17) > =09at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefore= s.java:26) > =09at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.= java:27) > =09at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > =09at org.junit.rules.RunRules.evaluate(RunRules.java:20) > =09at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > =09at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassR= unner.java:70) > =09at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassR= unner.java:50) > =09at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > =09at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > =09at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > =09at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > =09at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > =09at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefore= s.java:26) > =09at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.= java:27) > =09at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > =09at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provi= der.java:365) > =09at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JU= nit4Provider.java:273) > =09at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUni= t4Provider.java:238) > =09at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provid= er.java:159) > =09at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameC= lassLoader(ForkedBooter.java:379) > =09at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Fo= rkedBooter.java:340) > =09at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.= java:125) > =09at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.jav= a:413) > {code} > https://builds.apache.org/job/PreCommit-HIVE-Build/17591/testReport/org.a= pache.hadoop.hive.ql.parse/TestReplicationScenariosAcidTablesBootstrap/test= BootstrapAcidTablesDuringIncrementalWithConcurrentWrites/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)