hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mungeol Heo <mungeol....@gmail.com>
Subject [HDP][YARN] app timeline server stops with "too many open files" error.
Date Wed, 01 Jun 2016 03:48:42 GMT
Hi,

After starting app timeline server,
'/var/log/hadoop-yarn/yarn/yarn-yarn-timelineserver-hostname.net.log'
gives many message like below.

...
2016-06-01 12:36:28,641 INFO  timeline.RollingLevelDB
(RollingLevelDB.java:initRollingLevelDB(266)) - Added rolling leveldb
instance 2016-05-23-08 to indexes-ldb
2016-06-01 12:36:28,641 INFO  timeline.RollingLevelDB
(RollingLevelDB.java:initRollingLevelDB(258)) - Initializing rolling
leveldb instance
:file:/data/hadoop/yarn/timeline/leveldb-timeline-store/indexes-ldb.2016-05-09-14
for start time: 1462802400000
...

Then, it gives 'too mnay open files' error.

...
2016-06-01 12:36:28,715 INFO  service.AbstractService
(AbstractService.java:noteFailure(272)) - Service
org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore
failed in state INITED; cause:
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
/data/hadoop/yarn/timeline/leveldb-timeline-store/starttime-ldb/LOCK:
Too many open files
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
/data/hadoop/yarn/timeline/leveldb-timeline-store/starttime-ldb/LOCK:
Too many open files
        at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
        at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
        at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
        at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:324)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:151)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)
...

I've tired to increase nofile from 65000 to 650000 at
'/etc/security/limits.conf'.
And also increased 'yarn_user_nofile_limit', which is a yarn config,
value to 65536.
However, nothing works.

Any help will be grate.
Thank you.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message