kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@apache.org
Subject [1/2] kudu git commit: rpc: hook up a callback for libev fatal errors
Date Thu, 10 Aug 2017 19:01:04 GMT
Repository: kudu
Updated Branches:
  refs/heads/master ecd67486b -> 6de378296


rpc: hook up a callback for libev fatal errors

In troubleshooting a recent cluster issue, I found that the daemon had
run out of file descriptors. This caused libev to abort(), but the error
message wasn't anywhere obvious since the default implementation just
writes to stderr.

Piping this through to a GLog FATAL is more likely to result in an
obvious log message.

It's difficult to write an automated test for this, but I tested by
setting my ulimit to 10 and running rpc-test. This resulted in:

F0809 19:03:39.882194  3358 reactor.cc:108] LibEV fatal error: (libev)
error creating signal/async pipe: Too many open files [24]

Change-Id: I5fa77237a40f43d6bb82e9f1ceecd31d52268f9d
Reviewed-on: http://gerrit.cloudera.org:8080/7633
Tested-by: Kudu Jenkins
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Reviewed-by: David Ribeiro Alves <davidralves@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/2a99bb3e
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/2a99bb3e
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/2a99bb3e

Branch: refs/heads/master
Commit: 2a99bb3e5864ae4ae4dd6d2dcdff557fed81aa1d
Parents: ecd6748
Author: Todd Lipcon <todd@apache.org>
Authored: Wed Aug 9 19:01:07 2017 -0700
Committer: Todd Lipcon <todd@apache.org>
Committed: Thu Aug 10 18:47:58 2017 +0000

----------------------------------------------------------------------
 src/kudu/rpc/reactor.cc | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/2a99bb3e/src/kudu/rpc/reactor.cc
----------------------------------------------------------------------
diff --git a/src/kudu/rpc/reactor.cc b/src/kudu/rpc/reactor.cc
index cf63672..b3b7ea2 100644
--- a/src/kudu/rpc/reactor.cc
+++ b/src/kudu/rpc/reactor.cc
@@ -96,6 +96,22 @@ Status ShutdownError(bool aborted) {
       Status::Aborted(msg, "", ESHUTDOWN) :
       Status::ServiceUnavailable(msg, "", ESHUTDOWN);
 }
+
+// Callback for libev fatal errors (eg running out of file descriptors).
+// Unfortunately libev doesn't plumb these back through to the caller, but
+// instead just expects the callback to abort.
+//
+// This implementation is slightly preferable to the built-in one since
+// it uses a FATAL log message instead of printing to stderr, which might
+// not end up anywhere useful in a daemonized context.
+void LibevSysErr(const char* msg) throw() {
+  PLOG(FATAL) << "LibEV fatal error: " << msg;
+}
+
+void DoInitLibEv() {
+  ev::set_syserr_cb(LibevSysErr);
+}
+
 } // anonymous namespace
 
 ReactorThread::ReactorThread(Reactor *reactor, const MessengerBuilder& bld)
@@ -620,6 +636,8 @@ Reactor::Reactor(shared_ptr<Messenger> messenger,
       name_(StringPrintf("%s_R%03d", messenger_->name().c_str(), index)),
       closing_(false),
       thread_(this, bld) {
+  static std::once_flag libev_once;
+  std::call_once(libev_once, DoInitLibEv);
 }
 
 Status Reactor::Init() {


Mime
View raw message