hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric III <yaz...@gmail.com>
Subject Re: Re: [HDP][YARN] app timeline server stops with "too many open files" error.
Date Thu, 02 Jun 2016 16:06:17 GMT
Hi,

I'm having the same issue.
I've tried setting ulimit -n in yarn-daemon.sh, it does not seem to work.
Jon, What do you mean when you say "check out the ulimit -n" ?

Thanks,

Eric

> Modifying /etc/security/limits.conf won't take immediate effect.
> Either restart the system or check out the ulimit -n command and try
> again.

> Thanks,

> Jon

On Tue, May 31, 2016 at 11:48 PM, Mungeol Heo <mungeol.heo@gmail.com> wrote:
> Hi,
>
> After starting app timeline server,
> '/var/log/hadoop-yarn/yarn/yarn-yarn-timelineserver-hostname.net.log'
> gives many message like below.
>
> ...
> 2016-06-01 12:36:28,641 INFO  timeline.RollingLevelDB
> (RollingLevelDB.java:initRollingLevelDB(266)) - Added rolling leveldb
> instance 2016-05-23-08 to indexes-ldb
> 2016-06-01 12:36:28,641 INFO  timeline.RollingLevelDB
> (RollingLevelDB.java:initRollingLevelDB(258)) - Initializing rolling
> leveldb instance
> :file:/data/hadoop/yarn/timeline/leveldb-timeline-store/indexes-ldb.2016-05-09-14
> for start time: 1462802400000
> ...
>
> Then, it gives 'too mnay open files' error.
>
> ...
> 2016-06-01 12:36:28,715 INFO  service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service
> org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore
> failed in state INITED; cause:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /data/hadoop/yarn/timeline/leveldb-timeline-store/starttime-ldb/LOCK:
> Too many open files
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /data/hadoop/yarn/timeline/leveldb-timeline-store/starttime-ldb/LOCK:
> Too many open files
>         at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
>         at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
>         at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
>         at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:324)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:151)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)
> ...
>
> I've tired to increase nofile from 65000 to 650000 at
> '/etc/security/limits.conf'.
> And also increased 'yarn_user_nofile_limit', which is a yarn config,
> value to 65536.
> However, nothing works.
>
> Any help will be grate.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.orgEither restart the system or check
out the ulimit -n command and try
again.

Thanks,

Jon

On Tue, May 31, 2016 at 11:48 PM, Mungeol Heo <mungeol.heo@gmail.com> wrote:
> Hi,
>
> After starting app timeline server,
> '/var/log/hadoop-yarn/yarn/yarn-yarn-timelineserver-hostname.net.log'
> gives many message like below.
>
> ...
> 2016-06-01 12:36:28,641 INFO  timeline.RollingLevelDB
> (RollingLevelDB.java:initRollingLevelDB(266)) - Added rolling leveldb
> instance 2016-05-23-08 to indexes-ldb
> 2016-06-01 12:36:28,641 INFO  timeline.RollingLevelDB
> (RollingLevelDB.java:initRollingLevelDB(258)) - Initializing rolling
> leveldb instance
> :file:/data/hadoop/yarn/timeline/leveldb-timeline-store/indexes-ldb.2016-05-09-14
> for start time: 1462802400000
> ...
>
> Then, it gives 'too mnay open files' error.
>
> ...
> 2016-06-01 12:36:28,715 INFO  service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service
> org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore
> failed in state INITED; cause:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /data/hadoop/yarn/timeline/leveldb-timeline-store/starttime-ldb/LOCK:
> Too many open files
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /data/hadoop/yarn/timeline/leveldb-timeline-store/starttime-ldb/LOCK:
> Too many open files
>         at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
>         at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
>         at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
>         at org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.serviceInit(RollingLevelDBTimelineStore.java:324)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:151)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)
> ...
>
> I've tired to increase nofile from 65000 to 650000 at
> '/etc/security/limits.conf'.
> And also increased 'yarn_user_nofile_limit', which is a yarn config,
> value to 65536.
> However, nothing works.
>
> Any help will be grate.
> Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message