hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-12337) Ozone: Concurrent RocksDB open calls fail because of "No locks available"
Date Tue, 22 Aug 2017 13:44:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-12337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mukul Kumar Singh updated HDFS-12337:
-------------------------------------
    Status: Patch Available  (was: Open)

> Ozone: Concurrent RocksDB open calls fail because of "No locks available"
> -------------------------------------------------------------------------
>
>                 Key: HDFS-12337
>                 URL: https://issues.apache.org/jira/browse/HDFS-12337
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>             Fix For: HDFS-7240
>
>         Attachments: HDFS-12337-HDFS-7240.001.patch
>
>
> HDFS-12216 fixes the issue with the static container cache and re-using the same container
port on datanode restart. However TestKeys still fails after HDFS-12216 is fixed.
> The test is now failing because Concurrent RocksDB open calls fail. In the current code
BlockDeleting service and Dispatcher tries to open the db concurrently.
> This jira will also fix the keepPort property for Ratis Container port and set the reuse
address property for XceiverServerRatis correctly.
> {code}
> 2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO  utils.RocksDBStore (RocksDBStore.java:<init>(64))
- opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO  logging.LoggingHandler (Slf4JLogger.java:info(101))
- [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: [id: 0x61367e6f, /127.0.0.1:57173 => /127.0.0
> .1:57044]
> 2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore (RocksDBStore.java:<init>(64))
- opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
> 2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO  utils.RocksDBStore (RocksDBStore.java:<init>(67))
- Failed init RocksDB, db path : /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception
org.rocksdb.RocksDBException: lock /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK:
No locks available
> 2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO  background.BlockDeletingService
(BlockDeletingService.java:call(203))      - The elapsed time of task@70a576ee for deleting
blocks: 12ms.
> 2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO  impl.Dispatcher (ContainerUtils.java:logAndReturnError(129))
     - Operation: GetKey : Trace ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message: 621a3b15-b9fc-4d49-a6e6-29d4c40cc91f
: Result: UNABLE_TO_READ_METADATA_DB
> 2017-08-22 16:51:34,475 [Thread-382] INFO  exceptions.OzoneExceptionMapper (OzoneExceptionMapper.java:toResponse(39))
ozone  c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548
hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex: {"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"}
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message