hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Shelukhin <ser...@hortonworks.com>
Subject HiveQA machines are low on disk space
Date Mon, 25 Jan 2016 21:37:34 GMT
Hi.

Recently, there has occasionally been a stuck mini cluster test, usually
MiniTez, I think I’ve seen MiniSpark too.
Upon looking at the logs for one recent timeout, I don’t see any obvious
problem with Hive, but there’s this:

2016-01-25T09:23:15,658 WARN
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonito
r@6845113[]]: namenode.FSNamesystem (FSNamesystem.java:run(5159)) -
NameNode low on available disk space. Entering safe mode.
2016-01-25T09:23:15,658 INFO
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonito
r@6845113[]]: hdfs.StateChange (FSNamesystem.java:reportStatus(6003)) -
STATE* Safe mode is ON.
Resources are low on NN. Please add or free up more resources then turn
off safe mode manually. NOTE:  If you turn off safe mode before adding
resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin
-safemode leave" to turn safe mode off.



After that, there are many safe-mode related errors.


I think that could be related to recent changes to preserve more logs. Can
we perhaps scale down log preservation a little bit, or gzip them?
It could also be something else on HiveQA machine.


Mime
View raw message