Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C1865166DD6 for ; Tue, 22 Aug 2017 14:50:07 +0200 (CEST) Received: (qmail 14037 invoked by uid 500); 22 Aug 2017 12:50:03 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 14026 invoked by uid 99); 22 Aug 2017 12:50:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Aug 2017 12:50:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 40E9EC006E for ; Tue, 22 Aug 2017 12:50:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id sXdRZ10QrDuw for ; Tue, 22 Aug 2017 12:50:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 521775FE66 for ; Tue, 22 Aug 2017 12:50:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 859D1E00A3 for ; Tue, 22 Aug 2017 12:50:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 436912537F for ; Tue, 22 Aug 2017 12:50:00 +0000 (UTC) Date: Tue, 22 Aug 2017 12:50:00 +0000 (UTC) From: "Mukul Kumar Singh (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HDFS-12337) Ozone: Concurrent RocksDB open calls fail because of "No locks available" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Mukul Kumar Singh created HDFS-12337: ---------------------------------------- Summary: Ozone: Concurrent RocksDB open calls fail because of "No locks available" Key: HDFS-12337 URL: https://issues.apache.org/jira/browse/HDFS-12337 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Fix For: HDFS-7240 HDFS-12216 fixes the issue with the static container cache and re-using the same container port on datanode restart. However TestKeys still fails after HDFS-12216 is fixed. The test is now failing because Concurrent RocksDB open calls fail. In the current code BlockDeleting service and Dispatcher tries to open the db concurrently. This jira will also fix the keepPort property for Ratis Container port and set the reuse address property for XceiverServerRatis correctly. {code} 2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO utils.RocksDBStore (RocksDBStore.java:(64)) - opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db 2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO logging.LoggingHandler (Slf4JLogger.java:info(101)) - [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: [id: 0x61367e6f, /127.0.0.1:57173 => /127.0.0 .1:57044] 2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO utils.RocksDBStore (RocksDBStore.java:(64)) - opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db 2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO utils.RocksDBStore (RocksDBStore.java:(67)) - Failed init RocksDB, db path : /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception org.rocksdb.RocksDBException: lock /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK: No locks available 2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO background.BlockDeletingService (BlockDeletingService.java:call(203)) - The elapsed time of task@70a576ee for deleting blocks: 12ms. 2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO impl.Dispatcher (ContainerUtils.java:logAndReturnError(129)) - Operation: GetKey : Trace ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message: 621a3b15-b9fc-4d49-a6e6-29d4c40cc91f : Result: UNABLE_TO_READ_METADATA_DB 2017-08-22 16:51:34,475 [Thread-382] INFO exceptions.OzoneExceptionMapper (OzoneExceptionMapper.java:toResponse(39)) ozone c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548 hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex: {"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"} {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org