kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mpe...@apache.org
Subject [3/6] kudu git commit: perf: improve string data generation performance
Date Fri, 23 Mar 2018 05:11:49 GMT
perf: improve string data generation performance

This changes the 'perf loadgen' tool to not use stringstream for
formatting a string. Instead, it reuses a single string buffer and uses
some functions from gutil to achieve the same output.

Tested with:
   build/latest/bin/kudu.orig perf loadgen -num-threads 16 \
        -num-rows-per-thread $[10*1000*1000] \
       -table-num-replicas 3 -table-num-buckets=100 \
        my-master

Before this fix:
  time total  : 130030 ms
    2204814.701333      task-clock (msec)         #   16.535 CPUs utilized
  Top profile lines:
    14.17%  kudu.orig        libstdc++.so.6.0.19  [.] std::locale::locale
     7.26%  kudu.orig        libstdc++.so.6.0.19  [.] std::locale::~locale
     5.91%  kudu.orig        libstdc++.so.6.0.19  [.] std::use_facet<std::num_put<char,
std::ostreambuf_iterator<char, std::char_traits<char> > > >
     5.41%  kudu.orig        libstdc++.so.6.0.19  [.] std::has_facet<std::num_put<char,
std::ostreambuf_iterator<char, std::char_traits<char> > > >
     4.69%  kudu.orig        libstdc++.so.6.0.19  [.] std::has_facet<std::num_get<char,
std::istreambuf_iterator<char, std::char_traits<char> > > >

After this the throughput and CPU consumption were both improved by
about 40%:
  time total  : 90676.3 ms
    1458961.567442      task-clock (msec)         #   15.954 CPUs utilized

The top of the profile is now all related to client code instead of the
loadgen tool itself.

Change-Id: I4a3ebe39588d3da6a0775b168c5350fb420e025e
Reviewed-on: http://gerrit.cloudera.org:8080/9759
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/ffcadd5a
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/ffcadd5a
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/ffcadd5a

Branch: refs/heads/master
Commit: ffcadd5a60107c5e150c0083f1008a745e976280
Parents: 9ecf7d7
Author: Todd Lipcon <todd@apache.org>
Authored: Thu Mar 22 10:20:17 2018 -0700
Committer: Todd Lipcon <todd@apache.org>
Committed: Thu Mar 22 18:34:48 2018 +0000

----------------------------------------------------------------------
 src/kudu/tools/tool_action_perf.cc | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/ffcadd5a/src/kudu/tools/tool_action_perf.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tools/tool_action_perf.cc b/src/kudu/tools/tool_action_perf.cc
index 7c23bc8..2aeb514 100644
--- a/src/kudu/tools/tool_action_perf.cc
+++ b/src/kudu/tools/tool_action_perf.cc
@@ -135,6 +135,7 @@
 #include "kudu/gutil/map-util.h"
 #include "kudu/gutil/stl_util.h"
 #include "kudu/gutil/strings/split.h"
+#include "kudu/gutil/strings/strcat.h"
 #include "kudu/gutil/strings/substitute.h"
 #include "kudu/tools/tool_action_common.h"
 #include "kudu/util/decimal_util.h"
@@ -292,6 +293,7 @@ class Generator {
   uint64_t seq_;
   Random random_;
   const size_t string_len_;
+  string buf_;
 };
 
 template <>
@@ -311,15 +313,11 @@ float Generator::Next() {
 
 template <>
 string Generator::Next() {
-  ostringstream ss;
-  ss << NextImpl() << ".";
-  string str(ss.str());
-  if (str.size() >= string_len_) {
-    str = str.substr(0, string_len_);
-  } else {
-    str += string(string_len_ - str.size(), 'x');
-  }
-  return str;
+  buf_.clear();
+  StrAppend(&buf_, NextImpl(), ".");
+  // Truncate or extend with 'x's.
+  buf_.resize(string_len_, 'x');
+  return buf_;
 }
 
 Status GenerateRowData(Generator* gen, KuduPartialRow* row,


Mime
View raw message