Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BFD252009D9 for ; Wed, 18 May 2016 04:50:48 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BE47B160A23; Wed, 18 May 2016 02:50:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9F316160A24 for ; Wed, 18 May 2016 04:50:46 +0200 (CEST) Received: (qmail 55650 invoked by uid 500); 18 May 2016 02:50:45 -0000 Mailing-List: contact commits-help@hawq.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hawq.incubator.apache.org Delivered-To: mailing list commits@hawq.incubator.apache.org Received: (qmail 55641 invoked by uid 99); 18 May 2016 02:50:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2016 02:50:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 147E8C0FE0 for ; Wed, 18 May 2016 02:50:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.211 X-Spam-Level: X-Spam-Status: No, score=-3.211 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, T_FILL_THIS_FORM_SHORT=0.01] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 98fSMoJTtOP7 for ; Wed, 18 May 2016 02:50:27 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id 5CCB75FD07 for ; Wed, 18 May 2016 02:50:16 +0000 (UTC) Received: (qmail 53363 invoked by uid 99); 18 May 2016 02:50:14 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2016 02:50:14 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id D2FFADFC4F; Wed, 18 May 2016 02:50:14 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: rlei@apache.org To: commits@hawq.incubator.apache.org Date: Wed, 18 May 2016 02:50:32 -0000 Message-Id: <46ec902d72e14103a67382c9bd74e1a3@git.apache.org> In-Reply-To: <04ef34dcbc8745b3a15238fab69ea80a@git.apache.org> References: <04ef34dcbc8745b3a15238fab69ea80a@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [19/51] [abbrv] [partial] incubator-hawq git commit: HAWQ-735. Import thrift-0.9.3 into depends/thirdparty/thrift folder archived-at: Wed, 18 May 2016 02:50:48 -0000 http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.cpp ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.cpp b/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.cpp new file mode 100644 index 0000000..7a61b21 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.cpp @@ -0,0 +1,481 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#include + +#include "FacebookBase.h" +#include "ServiceTracker.h" +#include + +using namespace std; +using namespace facebook::fb303; +using namespace apache::thrift::concurrency; + + +uint64_t ServiceTracker::CHECKPOINT_MINIMUM_INTERVAL_SECONDS = 60; +int ServiceTracker::LOG_LEVEL = 5; + + +ServiceTracker::ServiceTracker(facebook::fb303::FacebookBase *handler, + void (*logMethod)(int, const string &), + bool featureCheckpoint, + bool featureStatusCheck, + bool featureThreadCheck, + Stopwatch::Unit stopwatchUnit) + : handler_(handler), logMethod_(logMethod), + featureCheckpoint_(featureCheckpoint), + featureStatusCheck_(featureStatusCheck), + featureThreadCheck_(featureThreadCheck), + stopwatchUnit_(stopwatchUnit), + checkpointServices_(0) +{ + if (featureCheckpoint_) { + time_t now = time(NULL); + checkpointTime_ = now; + } else { + checkpointTime_ = 0; + } +} + +/** + * Registers the beginning of a "service method": basically, any of + * the implementations of Thrift remote procedure calls that a + * FacebookBase handler is handling. Controls concurrent + * services and reports statistics (via log and via fb303 counters). + * Throws an exception if the server is not ready to handle service + * methods yet. + * + * note: The relationship between startService() and finishService() + * is currently defined so that a call to finishService() should only + * be matched to this call to startService() if this method returns + * without exception. It wouldn't be a problem to implement things + * the other way, so that *every* start needed a finish, but this + * convention was chosen to match the way an object's constructor and + * destructor work together, i.e. to work well with ServiceMethod + * objects. + * + * @param const ServiceMethod &serviceMethod A reference to the ServiceMethod + * object instantiated at the start + * of the service method. + */ +void +ServiceTracker::startService(const ServiceMethod &serviceMethod) +{ + // note: serviceMethod.timer_ automatically starts at construction. + + // log service start + logMethod_(5, serviceMethod.signature_); + + // check handler ready + if (featureStatusCheck_ && !serviceMethod.featureLogOnly_) { + // note: Throwing exceptions before counting statistics. See note + // in method header. + // note: A STOPPING server is not accepting new connections, but it + // is still handling any already-connected threads -- so from the + // service method's point of view, a status of STOPPING is a green + // light. + facebook::fb303::fb_status status = handler_->getStatus(); + if (status != facebook::fb303::ALIVE + && status != facebook::fb303::STOPPING) { + if (status == facebook::fb303::STARTING) { + throw ServiceException("Server starting up; please try again later"); + } else { + throw ServiceException("Server not alive; please try again later"); + } + } + } + + // check server threads + if (featureThreadCheck_ && !serviceMethod.featureLogOnly_) { + // note: Might want to put these messages in reportCheckpoint() if + // log is getting spammed. + if (threadManager_ != NULL) { + size_t idle_count = threadManager_->idleWorkerCount(); + if (idle_count == 0) { + stringstream message; + message << "service " << serviceMethod.signature_ + << ": all threads (" << threadManager_->workerCount() + << ") in use"; + logMethod_(3, message.str()); + } + } + } +} + +/** + * Logs a significant step in the middle of a "service method"; see + * startService. + * + * @param const ServiceMethod &serviceMethod A reference to the ServiceMethod + * object instantiated at the start + * of the service method. + * @return int64_t Elapsed units (see stopwatchUnit_) since ServiceMethod + * instantiation. + */ +int64_t +ServiceTracker::stepService(const ServiceMethod &serviceMethod, + const string &stepName) +{ + stringstream message; + string elapsed_label; + int64_t elapsed = serviceMethod.timer_.elapsedUnits(stopwatchUnit_, + &elapsed_label); + message << serviceMethod.signature_ + << ' ' << stepName + << " [" << elapsed_label << ']'; + logMethod_(5, message.str()); + return elapsed; +} + +/** + * Registers the end of a "service method"; see startService(). + * + * @param const ServiceMethod &serviceMethod A reference to the ServiceMethod + * object instantiated at the start + * of the service method. + */ +void +ServiceTracker::finishService(const ServiceMethod &serviceMethod) +{ + // log end of service + stringstream message; + string duration_label; + int64_t duration = serviceMethod.timer_.elapsedUnits(stopwatchUnit_, + &duration_label); + message << serviceMethod.signature_ + << " finish [" << duration_label << ']'; + logMethod_(5, message.str()); + + // count, record, and maybe report service statistics + if (!serviceMethod.featureLogOnly_) { + + if (!featureCheckpoint_) { + + // lifetime counters + // (note: No need to lock statisticsMutex_ if not doing checkpoint; + // FacebookService::incrementCounter() is already thread-safe.) + handler_->incrementCounter("lifetime_services"); + + } else { + + statisticsMutex_.lock(); + // note: No exceptions expected from this code block. Wrap in a try + // just to be safe. + try { + + // lifetime counters + // note: Good to synchronize this with the increment of + // checkpoint services, even though incrementCounter() is + // already thread-safe, for the sake of checkpoint reporting + // consistency (i.e. since the last checkpoint, + // lifetime_services has incremented by checkpointServices_). + handler_->incrementCounter("lifetime_services"); + + // checkpoint counters + checkpointServices_++; + checkpointDuration_ += duration; + + // per-service timing + // note kjv: According to my tests it is very slightly faster to + // call insert() once (and detect not-found) than calling find() + // and then maybe insert (if not-found). However, the difference + // is tiny for small maps like this one, and the code for the + // faster solution is slightly less readable. Also, I wonder if + // the instantiation of the (often unused) pair to insert makes + // the first algorithm slower after all. + map >::iterator iter; + iter = checkpointServiceDuration_.find(serviceMethod.name_); + if (iter != checkpointServiceDuration_.end()) { + iter->second.first++; + iter->second.second += duration; + } else { + checkpointServiceDuration_.insert(make_pair(serviceMethod.name_, + make_pair(1, duration))); + } + + // maybe report checkpoint + // note: ...if it's been long enough since the last report. + time_t now = time(NULL); + uint64_t check_interval = now - checkpointTime_; + if (check_interval >= CHECKPOINT_MINIMUM_INTERVAL_SECONDS) { + reportCheckpoint(); + } + + } catch (...) { + statisticsMutex_.unlock(); + throw; + } + statisticsMutex_.unlock(); + + } + } +} + +/** + * Logs some statistics gathered since the last call to this method. + * + * note: Thread race conditions on this method could cause + * misreporting and/or undefined behavior; the caller must protect + * uses of the object variables (and calls to this method) with a + * mutex. + * + */ +void +ServiceTracker::reportCheckpoint() +{ + time_t now = time(NULL); + + uint64_t check_count = checkpointServices_; + uint64_t check_interval = now - checkpointTime_; + uint64_t check_duration = checkpointDuration_; + + // export counters for timing of service methods (by service name) + handler_->setCounter("checkpoint_time", check_interval); + map >::iterator iter; + uint64_t count; + for (iter = checkpointServiceDuration_.begin(); + iter != checkpointServiceDuration_.end(); + ++iter) { + count = iter->second.first; + handler_->setCounter(string("checkpoint_count_") + iter->first, count); + if (count == 0) { + handler_->setCounter(string("checkpoint_speed_") + iter->first, + 0); + } else { + handler_->setCounter(string("checkpoint_speed_") + iter->first, + iter->second.second / count); + } + } + + // reset checkpoint variables + // note: Clearing the map while other threads are using it might + // cause undefined behavior. + checkpointServiceDuration_.clear(); + checkpointTime_ = now; + checkpointServices_ = 0; + checkpointDuration_ = 0; + + // get lifetime variables + uint64_t life_count = handler_->getCounter("lifetime_services"); + uint64_t life_interval = now - handler_->aliveSince(); + + // log checkpoint + stringstream message; + message << "checkpoint_time:" << check_interval + << " checkpoint_services:" << check_count + << " checkpoint_speed_sum:" << check_duration + << " lifetime_time:" << life_interval + << " lifetime_services:" << life_count; + if (featureThreadCheck_ && threadManager_ != NULL) { + size_t worker_count = threadManager_->workerCount(); + size_t idle_count = threadManager_->idleWorkerCount(); + message << " total_workers:" << worker_count + << " active_workers:" << (worker_count - idle_count); + } + logMethod_(4, message.str()); +} + +/** + * Remembers the thread manager used in the server, for monitoring thread + * activity. + * + * @param shared_ptr threadManager The server's thread manager. + */ +void +ServiceTracker::setThreadManager(boost::shared_ptr + threadManager) +{ + threadManager_ = threadManager; +} + +/** + * Logs messages to stdout; the passed message will be logged if the + * passed level is less than or equal to LOG_LEVEL. + * + * This is the default logging method used by the ServiceTracker. An + * alternate logging method (that accepts the same parameters) may be + * specified to the constructor. + * + * @param int level A level associated with the message: higher levels + * are used to indicate higher levels of detail. + * @param string message The message to log. + */ +void +ServiceTracker::defaultLogMethod(int level, const string &message) +{ + if (level <= LOG_LEVEL) { + string level_string; + time_t now = time(NULL); + char now_pretty[26]; + ctime_r(&now, now_pretty); + now_pretty[24] = '\0'; + switch (level) { + case 1: + level_string = "CRITICAL"; + break; + case 2: + level_string = "ERROR"; + break; + case 3: + level_string = "WARNING"; + break; + case 5: + level_string = "DEBUG"; + break; + case 4: + default: + level_string = "INFO"; + break; + } + cout << '[' << level_string << "] [" << now_pretty << "] " + << message << endl; + } +} + + +/** + * Creates a Stopwatch, which can report the time elapsed since its + * creation. + * + */ +Stopwatch::Stopwatch() +{ + gettimeofday(&startTime_, NULL); +} + +void +Stopwatch::reset() +{ + gettimeofday(&startTime_, NULL); +} + +uint64_t +Stopwatch::elapsedUnits(Stopwatch::Unit unit, string *label) const +{ + timeval now_time; + gettimeofday(&now_time, NULL); + time_t duration_secs = now_time.tv_sec - startTime_.tv_sec; + + uint64_t duration_units; + switch (unit) { + case UNIT_SECONDS: + duration_units = duration_secs + + (now_time.tv_usec - startTime_.tv_usec + 500000) / 1000000; + if (NULL != label) { + stringstream ss_label; + ss_label << duration_units << " secs"; + label->assign(ss_label.str()); + } + break; + case UNIT_MICROSECONDS: + duration_units = duration_secs * 1000000 + + now_time.tv_usec - startTime_.tv_usec; + if (NULL != label) { + stringstream ss_label; + ss_label << duration_units << " us"; + label->assign(ss_label.str()); + } + break; + case UNIT_MILLISECONDS: + default: + duration_units = duration_secs * 1000 + + (now_time.tv_usec - startTime_.tv_usec + 500) / 1000; + if (NULL != label) { + stringstream ss_label; + ss_label << duration_units << " ms"; + label->assign(ss_label.str()); + } + break; + } + return duration_units; +} + +/** + * Creates a ServiceMethod, used for tracking a single service method + * invocation (via the ServiceTracker). The passed name of the + * ServiceMethod is used to group statistics (e.g. counts and durations) + * for similar invocations; the passed signature is used to uniquely + * identify the particular invocation in the log. + * + * note: A version of this constructor is provided that automatically + * forms a signature the name and a passed numeric id. Silly, sure, + * but commonly used, since it often saves the caller a line or two of + * code. + * + * @param ServiceTracker *tracker The service tracker that will track this + * ServiceMethod. + * @param const string &name The service method name (usually independent + * of service method parameters). + * @param const string &signature A signature uniquely identifying the method + * invocation (usually name plus parameters). + */ +ServiceMethod::ServiceMethod(ServiceTracker *tracker, + const string &name, + const string &signature, + bool featureLogOnly) + : tracker_(tracker), name_(name), signature_(signature), + featureLogOnly_(featureLogOnly) +{ + // note: timer_ automatically starts at construction. + + // invoke tracker to start service + // note: Might throw. If it throws, then this object's destructor + // won't be called, which is according to plan: finishService() is + // only supposed to be matched to startService() if startService() + // returns without error. + tracker_->startService(*this); +} + +ServiceMethod::ServiceMethod(ServiceTracker *tracker, + const string &name, + uint64_t id, + bool featureLogOnly) + : tracker_(tracker), name_(name), featureLogOnly_(featureLogOnly) +{ + // note: timer_ automatically starts at construction. + stringstream ss_signature; + ss_signature << name << " (" << id << ')'; + signature_ = ss_signature.str(); + + // invoke tracker to start service + // note: Might throw. If it throws, then this object's destructor + // won't be called, which is according to plan: finishService() is + // only supposed to be matched to startService() if startService() + // returns without error. + tracker_->startService(*this); +} + +ServiceMethod::~ServiceMethod() +{ + // invoke tracker to finish service + // note: Not expecting an exception from this code, but + // finishService() might conceivably throw an out-of-memory + // exception. + try { + tracker_->finishService(*this); + } catch (...) { + // don't throw + } +} + +uint64_t +ServiceMethod::step(const std::string &stepName) +{ + return tracker_->stepService(*this, stepName); +} http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.h ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.h b/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.h new file mode 100644 index 0000000..9a3edd8 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/cpp/ServiceTracker.h @@ -0,0 +1,215 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * ServiceTracker is a utility class for logging and timing service + * calls to a fb303 Thrift server. Currently, ServiceTracker offers + * the following features: + * + * . Logging of service method start, end (and duration), and + * optional steps in between. + * + * . Automatic check of server status via fb303::getStatus() + * with a ServiceException thrown if server not alive + * (at method start). + * + * . A periodic logged checkpoint reporting lifetime time, lifetime + * service count, and per-method statistics since the last checkpoint + * time (at method finish). + * + * . Export of fb303 counters for lifetime and checkpoint statistics + * (at method finish). + * + * . For TThreadPoolServers, a logged warning when all server threads + * are busy (at method start). (Must call setThreadManager() after + * ServiceTracker instantiation for this feature to be enabled.) + * + * Individual features may be enabled or disabled by arguments to the + * constructor. The constructor also accepts a pointer to a logging + * method -- if no pointer is passed, the tracker will log to stdout. + * + * ServiceTracker defines private methods for service start, finish, + * and step, which are designed to be accessed by instantiating a + * friend ServiceMethod object, as in the following example: + * + * #include + * class MyServiceHandler : virtual public MyServiceIf, + * public facebook::fb303::FacebookBase + * { + * public: + * MyServiceHandler::MyServiceHandler() : mServiceTracker(this) {} + * void MyServiceHandler::myServiceMethod(int userId) { + * // note: Instantiating a ServiceMethod object starts a timer + * // and tells the ServiceTracker to log the start. Might throw + * // a ServiceException. + * ServiceMethod serviceMethod(&mServiceTracker, + * "myServiceMethod", + * userId); + * ... + * // note: Calling the step method tells the ServiceTracker to + * // log the step, with a time elapsed since start. + * serviceMethod.step("post parsing, begin processing"); + * ... + * // note: When the ServiceMethod object goes out of scope, the + * // ServiceTracker will log the total elapsed time of the method. + * } + * ... + * private: + * ServiceTracker mServiceTracker; + * } + * + * The step() method call is optional; the startService() and + * finishService() methods are handled by the object's constructor and + * destructor. + * + * The ServiceTracker is (intended to be) thread-safe. + * + * Future: + * + * . Come up with something better for logging than passing a + * function pointer to the constructor. + * + * . Add methods for tracking errors from service methods, e.g. + * ServiceTracker::reportService(). + */ + +#ifndef SERVICETRACKER_H +#define SERVICETRACKER_H + + +#include +#include +#include +#include +#include +#include + +#include + + +namespace apache { namespace thrift { namespace concurrency { + class ThreadManager; +}}} + + +namespace facebook { namespace fb303 { + + +class FacebookBase; +class ServiceMethod; + + +class Stopwatch +{ +public: + enum Unit { UNIT_SECONDS, UNIT_MILLISECONDS, UNIT_MICROSECONDS }; + Stopwatch(); + uint64_t elapsedUnits(Unit unit, std::string *label = NULL) const; + void reset(); +private: + timeval startTime_; +}; + + +class ServiceTracker +{ + friend class ServiceMethod; + +public: + + static uint64_t CHECKPOINT_MINIMUM_INTERVAL_SECONDS; + static int LOG_LEVEL; + + ServiceTracker(facebook::fb303::FacebookBase *handler, + void (*logMethod)(int, const std::string &) + = &ServiceTracker::defaultLogMethod, + bool featureCheckpoint = true, + bool featureStatusCheck = true, + bool featureThreadCheck = true, + Stopwatch::Unit stopwatchUnit + = Stopwatch::UNIT_MILLISECONDS); + + void setThreadManager(boost::shared_ptr threadManager); + +private: + + facebook::fb303::FacebookBase *handler_; + void (*logMethod_)(int, const std::string &); + boost::shared_ptr threadManager_; + + bool featureCheckpoint_; + bool featureStatusCheck_; + bool featureThreadCheck_; + Stopwatch::Unit stopwatchUnit_; + + apache::thrift::concurrency::Mutex statisticsMutex_; + time_t checkpointTime_; + uint64_t checkpointServices_; + uint64_t checkpointDuration_; + std::map > checkpointServiceDuration_; + + void startService(const ServiceMethod &serviceMethod); + int64_t stepService(const ServiceMethod &serviceMethod, + const std::string &stepName); + void finishService(const ServiceMethod &serviceMethod); + void reportCheckpoint(); + static void defaultLogMethod(int level, const std::string &message); +}; + + +class ServiceMethod +{ + friend class ServiceTracker; +public: + ServiceMethod(ServiceTracker *tracker, + const std::string &name, + const std::string &signature, + bool featureLogOnly = false); + ServiceMethod(ServiceTracker *tracker, + const std::string &name, + uint64_t id, + bool featureLogOnly = false); + ~ServiceMethod(); + uint64_t step(const std::string &stepName); +private: + ServiceTracker *tracker_; + std::string name_; + std::string signature_; + bool featureLogOnly_; + Stopwatch timer_; +}; + + +class ServiceException : public std::exception +{ +public: + explicit ServiceException(const std::string &message, int code = 0) + : message_(message), code_(code) {} + ~ServiceException() throw() {} + virtual const char *what() const throw() { return message_.c_str(); } + int code() const throw() { return code_; } +private: + std::string message_; + int code_; +}; + + +}} // facebook::fb303 + +#endif http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/global_footer.mk ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/global_footer.mk b/depends/thirdparty/thrift/contrib/fb303/global_footer.mk new file mode 100644 index 0000000..96f82eb --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/global_footer.mk @@ -0,0 +1,21 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +thriftstyle : $(XBUILT_SOURCES) + http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/global_header.mk ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/global_header.mk b/depends/thirdparty/thrift/contrib/fb303/global_header.mk new file mode 100644 index 0000000..77c9455 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/global_header.mk @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +#define thrift_template +# $(1) : $(2) +# $$(THRIFT) $(3) $(4) $(5) $(6) $(7) $(8) $$< +#endef + +define thrift_template +XTARGET := $(shell perl -e '@val = split("\/","$(2)"); $$last = pop(@val);split("\\.",$$last);print "$(1)/"."gen-cpp/"."@_[0]"."_types.cpp\n"' ) + +ifneq ($$(XBUILT_SOURCES),) + XBUILT_SOURCES := $$(XBUILT_SOURCES) $$(XTARGET) +else + XBUILT_SOURCES := $$(XTARGET) +endif +$$(XTARGET) : $(2) + $$(THRIFT) -o $1 $3 $$< +endef + +clean-common: + rm -rf gen-* http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/if/fb303.thrift ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/if/fb303.thrift b/depends/thirdparty/thrift/contrib/fb303/if/fb303.thrift new file mode 100644 index 0000000..66c8315 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/if/fb303.thrift @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * fb303.thrift + */ + +namespace java com.facebook.fb303 +namespace cpp facebook.fb303 +namespace perl Facebook.FB303 + +/** + * Common status reporting mechanism across all services + */ +enum fb_status { + DEAD = 0, + STARTING = 1, + ALIVE = 2, + STOPPING = 3, + STOPPED = 4, + WARNING = 5, +} + +/** + * Standard base service + */ +service FacebookService { + + /** + * Returns a descriptive name of the service + */ + string getName(), + + /** + * Returns the version of the service + */ + string getVersion(), + + /** + * Gets the status of this service + */ + fb_status getStatus(), + + /** + * User friendly description of status, such as why the service is in + * the dead or warning state, or what is being started or stopped. + */ + string getStatusDetails(), + + /** + * Gets the counters for this service + */ + map getCounters(), + + /** + * Gets the value of a single counter + */ + i64 getCounter(1: string key), + + /** + * Sets an option + */ + void setOption(1: string key, 2: string value), + + /** + * Gets an option + */ + string getOption(1: string key), + + /** + * Gets all options + */ + map getOptions(), + + /** + * Returns a CPU profile over the given time interval (client and server + * must agree on the profile format). + */ + string getCpuProfile(1: i32 profileDurationInSec), + + /** + * Returns the unix time that the server has been running since + */ + i64 aliveSince(), + + /** + * Tell the server to reload its configuration, reopen log files, etc + */ + oneway void reinitialize(), + + /** + * Suggest a shutdown to the server + */ + oneway void shutdown(), + +} http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/java/build.xml ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/java/build.xml b/depends/thirdparty/thrift/contrib/fb303/java/build.xml new file mode 100755 index 0000000..8f2fa51 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/java/build.xml @@ -0,0 +1,195 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/java/src/FacebookBase.java ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/java/src/FacebookBase.java b/depends/thirdparty/thrift/contrib/fb303/java/src/FacebookBase.java new file mode 100644 index 0000000..f8f26a3 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/java/src/FacebookBase.java @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package com.facebook.fb303; + +import java.util.AbstractMap; +import java.util.HashMap; +import java.util.concurrent.ConcurrentHashMap; + +public abstract class FacebookBase implements FacebookService.Iface { + + private String name_; + + private long alive_; + + private final ConcurrentHashMap counters_ = + new ConcurrentHashMap(); + + private final ConcurrentHashMap options_ = + new ConcurrentHashMap(); + + protected FacebookBase(String name) { + name_ = name; + alive_ = System.currentTimeMillis() / 1000; + } + + public String getName() { + return name_; + } + + public abstract fb_status getStatus(); + + public String getStatusDetails() { + return ""; + } + + public void deleteCounter(String key) { + counters_.remove(key); + } + + public void resetCounter(String key) { + counters_.put(key, 0L); + } + + public long incrementCounter(String key) { + long val = getCounter(key) + 1; + counters_.put(key, val); + return val; + } + + public long incrementCounter(String key, long increment) { + long val = getCounter(key) + increment; + counters_.put(key, val); + return val; + } + + public long setCounter(String key, long value) { + counters_.put(key, value); + return value; + } + + public AbstractMap getCounters() { + return counters_; + } + + public long getCounter(String key) { + Long val = counters_.get(key); + if (val == null) { + return 0; + } + return val.longValue(); + } + + public void setOption(String key, String value) { + options_.put(key, value); + } + + public String getOption(String key) { + return options_.get(key); + } + + public AbstractMap getOptions() { + return options_; + } + + public long aliveSince() { + return alive_; + } + + public String getCpuProfile() { + return ""; + } + + public void reinitialize() {} + + public void shutdown() {} + +} http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/php/FacebookBase.php ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/php/FacebookBase.php b/depends/thirdparty/thrift/contrib/fb303/php/FacebookBase.php new file mode 100644 index 0000000..2ac318f --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/php/FacebookBase.php @@ -0,0 +1,89 @@ +name_ = $name; + } + + public function getName() { + return $this->name_; + } + + public function getVersion() { + return ''; + } + + public function getStatus() { + return null; + } + + public function getStatusDetails() { + return ''; + } + + public function getCounters() { + return array(); + } + + public function getCounter($key) { + return null; + } + + public function setOption($key, $value) { + return; + } + + public function getOption($key) { + return ''; + } + + public function getOptions() { + return array(); + } + + public function aliveSince() { + return 0; + } + + public function getCpuProfile($duration) { + return ''; + } + + public function getLimitedReflection() { + return array(); + } + + public function reinitialize() { + return; + } + + public function shutdown() { + return; + } + +} + http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/py/Makefile.am ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/py/Makefile.am b/depends/thirdparty/thrift/contrib/fb303/py/Makefile.am new file mode 100644 index 0000000..060495e --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/py/Makefile.am @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +DESTDIR ?= / +EXTRA_DIST = setup.py src + +all: + +all-local: + $(thrift_home)/bin/thrift --gen py $(top_srcdir)/if/fb303.thrift + mv gen-py/fb303/* fb303 + $(PYTHON) setup.py build + +# We're ignoring prefix here because site-packages seems to be +# the equivalent of /usr/local/lib in Python land. +# Old version (can't put inline because it's not portable). +#$(PYTHON) setup.py install --prefix=$(prefix) --root=$(DESTDIR) $(PYTHON_SETUPUTIL_ARGS) +install-exec-hook: + $(PYTHON) setup.py install --root=$(DESTDIR) --prefix=$(PY_PREFIX) $(PYTHON_SETUPUTIL_ARGS) + + + +clean: clean-local + +clean-local: + $(RM) -r build + +check-local: all http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/py/fb303/FacebookBase.py ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/py/fb303/FacebookBase.py b/depends/thirdparty/thrift/contrib/fb303/py/fb303/FacebookBase.py new file mode 100644 index 0000000..685ff20 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/py/fb303/FacebookBase.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +import time +import FacebookService +import thrift.reflection.limited +from ttypes import fb_status + +class FacebookBase(FacebookService.Iface): + + def __init__(self, name): + self.name = name + self.alive = int(time.time()) + self.counters = {} + + def getName(self, ): + return self.name + + def getVersion(self, ): + return '' + + def getStatus(self, ): + return fb_status.ALIVE + + def getCounters(self): + return self.counters + + def resetCounter(self, key): + self.counters[key] = 0 + + def getCounter(self, key): + if self.counters.has_key(key): + return self.counters[key] + return 0 + + def incrementCounter(self, key): + self.counters[key] = self.getCounter(key) + 1 + + def setOption(self, key, value): + pass + + def getOption(self, key): + return "" + + def getOptions(self): + return {} + + def getOptions(self): + return {} + + def aliveSince(self): + return self.alive + + def getCpuProfile(self, duration): + return "" + + def getLimitedReflection(self): + return thrift.reflection.limited.Service() + + def reinitialize(self): + pass + + def shutdown(self): + pass http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/__init__.py ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/__init__.py b/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/__init__.py new file mode 100644 index 0000000..f8e3a94 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/__init__.py @@ -0,0 +1,20 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +__all__ = ['fb303_simple_mgmt'] http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/fb303_simple_mgmt.py ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/fb303_simple_mgmt.py b/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/fb303_simple_mgmt.py new file mode 100644 index 0000000..4f8ce99 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/py/fb303_scripts/fb303_simple_mgmt.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +import sys, os +from optparse import OptionParser + +from thrift.Thrift import * + +from thrift.transport import TSocket +from thrift.transport import TTransport +from thrift.protocol import TBinaryProtocol + +from fb303 import * +from fb303.ttypes import * + +def service_ctrl( + command, + port, + trans_factory = None, + prot_factory = None): + """ + service_ctrl is a generic function to execute standard fb303 functions + + @param command: one of stop, start, reload, status, counters, name, alive + @param port: service's port + @param trans_factory: TTransportFactory to use for obtaining a TTransport. Default is + TBufferedTransportFactory + @param prot_factory: TProtocolFactory to use for obtaining a TProtocol. Default is + TBinaryProtocolFactory + """ + + if command in ["status"]: + try: + status = fb303_wrapper('status', port, trans_factory, prot_factory) + status_details = fb303_wrapper('get_status_details', port, trans_factory, prot_factory) + + msg = fb_status_string(status) + if (len(status_details)): + msg += " - %s" % status_details + print msg + + if (status == fb_status.ALIVE): + return 2 + else: + return 3 + except: + print "Failed to get status" + return 3 + + # scalar commands + if command in ["version","alive","name"]: + try: + result = fb303_wrapper(command, port, trans_factory, prot_factory) + print result + return 0 + except: + print "failed to get ",command + return 3 + + # counters + if command in ["counters"]: + try: + counters = fb303_wrapper('counters', port, trans_factory, prot_factory) + for counter in counters: + print "%s: %d" % (counter, counters[counter]) + return 0 + except: + print "failed to get counters" + return 3 + + + # Only root should be able to run the following commands + if os.getuid() == 0: + # async commands + if command in ["stop","reload"] : + try: + fb303_wrapper(command, port, trans_factory, prot_factory) + return 0 + except: + print "failed to tell the service to ", command + return 3 + else: + if command in ["stop","reload"]: + print "root privileges are required to stop or reload the service." + return 4 + + print "The following commands are available:" + for command in ["counters","name","version","alive","status"]: + print "\t%s" % command + print "The following commands are available for users with root privileges:" + for command in ["stop","reload"]: + print "\t%s" % command + + + + return 0; + + +def fb303_wrapper(command, port, trans_factory = None, prot_factory = None): + sock = TSocket.TSocket('localhost', port) + + # use input transport factory if provided + if (trans_factory is None): + trans = TTransport.TBufferedTransport(sock) + else: + trans = trans_factory.getTransport(sock) + + # use input protocol factory if provided + if (prot_factory is None): + prot = TBinaryProtocol.TBinaryProtocol(trans) + else: + prot = prot_factory.getProtocol(trans) + + # initialize client and open transport + fb303_client = FacebookService.Client(prot, prot) + trans.open() + + if (command == 'reload'): + fb303_client.reinitialize() + + elif (command == 'stop'): + fb303_client.shutdown() + + elif (command == 'status'): + return fb303_client.getStatus() + + elif (command == 'version'): + return fb303_client.getVersion() + + elif (command == 'get_status_details'): + return fb303_client.getStatusDetails() + + elif (command == 'counters'): + return fb303_client.getCounters() + + elif (command == 'name'): + return fb303_client.getName() + + elif (command == 'alive'): + return fb303_client.aliveSince() + + trans.close() + + +def fb_status_string(status_enum): + if (status_enum == fb_status.DEAD): + return "DEAD" + if (status_enum == fb_status.STARTING): + return "STARTING" + if (status_enum == fb_status.ALIVE): + return "ALIVE" + if (status_enum == fb_status.STOPPING): + return "STOPPING" + if (status_enum == fb_status.STOPPED): + return "STOPPED" + if (status_enum == fb_status.WARNING): + return "WARNING" + + +def main(): + + # parse command line options + parser = OptionParser() + commands=["stop","counters","status","reload","version","name","alive"] + + parser.add_option("-c", "--command", dest="command", help="execute this API", + choices=commands, default="status") + parser.add_option("-p","--port",dest="port",help="the service's port", + default=9082) + + (options, args) = parser.parse_args() + status = service_ctrl(options.command, options.port) + sys.exit(status) + + +if __name__ == '__main__': + main() http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/fb303/py/setup.py ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/fb303/py/setup.py b/depends/thirdparty/thrift/contrib/fb303/py/setup.py new file mode 100644 index 0000000..6d42bd1 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/fb303/py/setup.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python + +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +import sys +try: + from setuptools import setup, Extension +except: + from distutils.core import setup, Extension, Command + +setup(name = 'thrift_fb303', + version = '0.9.3', + description = 'Python bindings for the Apache Thrift FB303', + author = ['Thrift Developers'], + author_email = ['dev@thrift.apache.org'], + url = 'http://thrift.apache.org', + license = 'Apache License 2.0', + packages = [ + 'fb303', + 'fb303_scripts', + ], + classifiers = [ + 'Development Status :: 5 - Production/Stable', + 'Environment :: Console', + 'Intended Audience :: Developers', + 'Programming Language :: Python', + 'Programming Language :: Python :: 2', + 'Topic :: Software Development :: Libraries', + 'Topic :: System :: Networking' + ], +) + http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/mingw-cross-compile.sh ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/mingw-cross-compile.sh b/depends/thirdparty/thrift/contrib/mingw-cross-compile.sh new file mode 100755 index 0000000..7ed5d47 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/mingw-cross-compile.sh @@ -0,0 +1,19 @@ +#!/bin/sh +set -e + +./configure \ + --disable-libs \ + --build=i686-pc-linux-gnu \ + --host=i586-mingw32msvc \ + CC=i586-mingw32msvc-gcc CXX=i586-mingw32msvc-g++ + +make + +# Check two locations to be compatible with libtool 1.5.26 or 2.2.6b. +if test -f compiler/cpp/.libs/thrift.exe ; then + i586-mingw32msvc-strip compiler/cpp/.libs/thrift.exe -o ./thrift.exe +else + i586-mingw32msvc-strip compiler/cpp/thrift.exe -o ./thrift.exe +fi +echo "Finished compiling with resulting exe" +ls -l ./thrift.exe http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/parse_profiling.py ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/parse_profiling.py b/depends/thirdparty/thrift/contrib/parse_profiling.py new file mode 100755 index 0000000..3d46fb8 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/parse_profiling.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +""" +This script can be used to make the output from +apache::thrift::profile_print_info() more human-readable. + +It translates each executable file name and address into the corresponding +source file name, line number, and function name. By default, it also +demangles C++ symbol names. +""" + +import optparse +import os +import re +import subprocess +import sys + + +class AddressInfo(object): + """ + A class to store information about a particular address in an object file. + """ + def __init__(self, obj_file, address): + self.objectFile = obj_file + self.address = address + self.sourceFile = None + self.sourceLine = None + self.function = None + + +g_addrs_by_filename = {} +def get_address(filename, address): + """ + Retrieve an AddressInfo object for the specified object file and address. + + Keeps a global list of AddressInfo objects. Two calls to get_address() + with the same filename and address will always return the same AddressInfo + object. + """ + global g_addrs_by_filename + try: + by_address = g_addrs_by_filename[filename] + except KeyError: + by_address = {} + g_addrs_by_filename[filename] = by_address + + try: + addr_info = by_address[address] + except KeyError: + addr_info = AddressInfo(filename, address) + by_address[address] = addr_info + return addr_info + + +def translate_file_addresses(filename, addresses, options): + """ + Use addr2line to look up information for the specified addresses. + All of the addresses must belong to the same object file. + """ + # Do nothing if we can't find the file + if not os.path.isfile(filename): + return + + args = ['addr2line'] + if options.printFunctions: + args.append('-f') + args.extend(['-e', filename]) + + proc = subprocess.Popen(args, stdin=subprocess.PIPE, + stdout=subprocess.PIPE) + for address in addresses: + assert address.objectFile == filename + proc.stdin.write(address.address + '\n') + + if options.printFunctions: + function = proc.stdout.readline() + function = function.strip() + if not function: + raise Exception('unexpected EOF from addr2line') + address.function = function + + file_and_line = proc.stdout.readline() + file_and_line = file_and_line.strip() + if not file_and_line: + raise Exception('unexpected EOF from addr2line') + idx = file_and_line.rfind(':') + if idx < 0: + msg = 'expected file and line number from addr2line; got %r' % \ + (file_and_line,) + msg += '\nfile=%r, address=%r' % (filename, address.address) + raise Exception(msg) + + address.sourceFile = file_and_line[:idx] + address.sourceLine = file_and_line[idx+1:] + + (remaining_out, cmd_err) = proc.communicate() + retcode = proc.wait() + if retcode != 0: + raise subprocess.CalledProcessError(retcode, args) + + +def lookup_addresses(options): + """ + Look up source file information for all of the addresses currently stored + in the global list of AddressInfo objects. + """ + global g_addrs_by_filename + for (file, addresses) in g_addrs_by_filename.items(): + translate_file_addresses(file, addresses.values(), options) + + +class Entry(object): + """ + An entry in the thrift profile output. + Contains a header line, and a backtrace. + """ + def __init__(self, header): + self.header = header + self.bt = [] + + def addFrame(self, filename, address): + # If libc was able to determine the symbols names, the filename + # argument will be of the form (+) + # So, strip off anything after the last '(' + idx = filename.rfind('(') + if idx >= 0: + filename = filename[:idx] + + addr = get_address(filename, address) + self.bt.append(addr) + + def write(self, f, options): + f.write(self.header) + f.write('\n') + n = 0 + for address in self.bt: + f.write(' #%-2d %s:%s\n' % (n, address.sourceFile, + address.sourceLine)) + n += 1 + if options.printFunctions: + if address.function: + f.write(' %s\n' % (address.function,)) + else: + f.write(' ??\n') + + +def process_file(in_file, out_file, options): + """ + Read thrift profile output from the specified input file, and print + prettier information on the output file. + """ + # + # A naive approach would be to read the input line by line, + # and each time we come to a filename and address, pass it to addr2line + # and print the resulting information. Unfortunately, addr2line can be + # quite slow, especially with large executables. + # + # This approach is much faster. We read in all of the input, storing + # the addresses in each file that need to be resolved. We then call + # addr2line just once for each file. This is much faster than calling + # addr2line once per address. + # + + virt_call_regex = re.compile(r'^\s*T_VIRTUAL_CALL: (\d+) calls on (.*):$') + gen_prot_regex = re.compile( + r'^\s*T_GENERIC_PROTOCOL: (\d+) calls to (.*) with a (.*):$') + bt_regex = re.compile(r'^\s*#(\d+)\s*(.*) \[(0x[0-9A-Za-z]+)\]$') + + # Parse all of the input, and store it as Entry objects + entries = [] + current_entry = None + while True: + line = in_file.readline() + if not line: + break + + if line == '\n' or line.startswith('Thrift virtual call info:'): + continue + + virt_call_match = virt_call_regex.match(line) + if virt_call_match: + num_calls = int(virt_call_match.group(1)) + type_name = virt_call_match.group(2) + if options.cxxfilt: + # Type names reported by typeid() are internal names. + # By default, c++filt doesn't demangle internal type names. + # (Some versions of c++filt have a "-t" option to enable this. + # Other versions don't have this argument, but demangle type + # names passed as an argument, but not on stdin.) + # + # If the output is being filtered through c++filt, prepend + # "_Z" to the type name to make it look like an external name. + type_name = '_Z' + type_name + header = 'T_VIRTUAL_CALL: %d calls on "%s"' % \ + (num_calls, type_name) + if current_entry is not None: + entries.append(current_entry) + current_entry = Entry(header) + continue + + gen_prot_match = gen_prot_regex.match(line) + if gen_prot_match: + num_calls = int(gen_prot_match.group(1)) + type_name1 = gen_prot_match.group(2) + type_name2 = gen_prot_match.group(3) + if options.cxxfilt: + type_name1 = '_Z' + type_name1 + type_name2 = '_Z' + type_name2 + header = 'T_GENERIC_PROTOCOL: %d calls to "%s" with a "%s"' % \ + (num_calls, type_name1, type_name2) + if current_entry is not None: + entries.append(current_entry) + current_entry = Entry(header) + continue + + bt_match = bt_regex.match(line) + if bt_match: + if current_entry is None: + raise Exception('found backtrace frame before entry header') + frame_num = int(bt_match.group(1)) + filename = bt_match.group(2) + address = bt_match.group(3) + current_entry.addFrame(filename, address) + continue + + raise Exception('unexpected line in input: %r' % (line,)) + + # Add the last entry we were processing to the list + if current_entry is not None: + entries.append(current_entry) + current_entry = None + + # Look up all of the addresses + lookup_addresses(options) + + # Print out the entries, now that the information has been translated + for entry in entries: + entry.write(out_file, options) + out_file.write('\n') + + +def start_cppfilt(): + (read_pipe, write_pipe) = os.pipe() + + # Fork. Run c++filt in the parent process, + # and then continue normal processing in the child. + pid = os.fork() + if pid == 0: + # child + os.dup2(write_pipe, sys.stdout.fileno()) + os.close(read_pipe) + os.close(write_pipe) + return + else: + # parent + os.dup2(read_pipe, sys.stdin.fileno()) + os.close(read_pipe) + os.close(write_pipe) + + cmd = ['c++filt'] + os.execvp(cmd[0], cmd) + + +def main(argv): + parser = optparse.OptionParser(usage='%prog [options] []') + parser.add_option('--no-functions', help='Don\'t print function names', + dest='printFunctions', action='store_false', + default=True) + parser.add_option('--no-demangle', + help='Don\'t demangle C++ symbol names', + dest='cxxfilt', action='store_false', + default=True) + + (options, args) = parser.parse_args(argv[1:]) + num_args = len(args) + if num_args == 0: + in_file = sys.stdin + elif num_args == 1: + in_file = open(argv[1], 'r') + else: + parser.print_usage(sys.stderr) + print >> sys.stderr, 'trailing arguments: %s' % (' '.join(args[1:],)) + return 1 + + if options.cxxfilt: + start_cppfilt() + + process_file(in_file, sys.stdout, options) + + +if __name__ == '__main__': + rc = main(sys.argv) + sys.exit(rc) http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/d709f67d/depends/thirdparty/thrift/contrib/thrift-maven-plugin/pom.xml ---------------------------------------------------------------------- diff --git a/depends/thirdparty/thrift/contrib/thrift-maven-plugin/pom.xml b/depends/thirdparty/thrift/contrib/thrift-maven-plugin/pom.xml new file mode 100644 index 0000000..b12a821 --- /dev/null +++ b/depends/thirdparty/thrift/contrib/thrift-maven-plugin/pom.xml @@ -0,0 +1,86 @@ + + + + + 4.0.0 + org.apache.thrift + thrift-maven-plugin + maven-plugin + thrift-maven-plugin + 0.9.3 + + + + org.apache.maven.plugins + maven-compiler-plugin + + 1.7 + 1.7 + + + + org.apache.maven.plugins + maven-surefire-plugin + 2.14.1 + + + ${thrift.compiler} + + + + + + + + junit + junit + 4.11 + test + + + org.apache.maven + maven-plugin-api + 3.1.0 + + + org.apache.maven + maven-project + 2.2.1 + + + com.google.guava + guava + 14.0.1 + + + org.codehaus.plexus + plexus-utils + 3.0.14 + + + + ${basedir}/../.. + ${thrift.root}/compiler/cpp/thrift + ${thrift.root}/test + +