hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HBASE-19025) Hbase MasterProcWALs getting pile up
Date Tue, 17 Oct 2017 20:00:01 GMT

     [ https://issues.apache.org/jira/browse/HBASE-19025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Elser resolved HBASE-19025.
--------------------------------
    Resolution: Invalid

Please leave JIRA issues for making code changes to HBase, and take questions like these to
the mailing lists.

Looks like you need to increase the number of open files allowed via ulimit on your datanodes.

> Hbase MasterProcWALs getting pile up
> ------------------------------------
>
>                 Key: HBASE-19025
>                 URL: https://issues.apache.org/jira/browse/HBASE-19025
>             Project: HBase
>          Issue Type: Bug
>          Components: hbase
>    Affects Versions: 1.2.6
>            Reporter: Irfan 
>
> MasterProcWALs getting pile up and getting below error in Hmaster log
> 2017-10-17 03:34:42,141 INFO  [ip-176-0-0-44:16000.activeMasterManager] hdfs.DFSClient:
Successfully connected to /176.0.0.29:50010 for BP-1665161556-176.0.
> 0.44-1498805562805:blk_1073773522_32746
> 2017-10-17 03:34:42,182 INFO  [ip-176-0-0-44:16000.activeMasterManager] util.FSHDFSUtils:
Recover lease on dfs file hdfs://192.168.168.1:9000/hbase2/MasterProcWALs/state-00000000000000005592.log
> 2017-10-17 03:34:42,182 INFO  [ip-176-0-0-44:16000.activeMasterManager] util.FSHDFSUtils:
Recovered lease, attempt=0 on file=hdfs://ec2-34-195-113-113.compu
> te-1.amazonaws.com:9000/hbase2/MasterProcWALs/state-00000000000000005592.log after 0ms
> 2017-10-17 03:34:42,184 WARN  [ip-176-0-0-44:16000.activeMasterManager] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
> java.io.IOException: Got error, status message opReadBlock BP-1665161556-176.0.0.44-1498805562805:blk_1073773523_32747
received exception java.io.FileNotFou
> ndException: /data/datanode/data/current/BP-1665161556-176.0.0.44-1498805562805/current/finalized/subdir0/subdir123/blk_1073773523_32747.meta
(Too many open
>  files), for OP_READ_BLOCK, self=/176.0.0.44:59340, remote=/176.0.0.44:50010, for file
/hbase2/MasterProcWALs/state-00000000000000005592.log, for pool BP-16
> 65161556-176.0.0.44-1498805562805 block 1073773523_32747



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message