Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C9DDA187A7 for ; Fri, 29 May 2015 11:53:38 +0000 (UTC) Received: (qmail 47172 invoked by uid 500); 29 May 2015 11:53:38 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 47136 invoked by uid 500); 29 May 2015 11:53:38 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 47122 invoked by uid 99); 29 May 2015 11:53:38 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 May 2015 11:53:38 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 72C161DDBCC; Fri, 29 May 2015 11:53:36 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============1888250300247286755==" MIME-Version: 1.0 Subject: Re: Review Request 34804: Work with json module takes too long From: "Andrew Onischuk" To: "Dmitro Lisnichenko" Cc: "Andrew Onischuk" , "Ambari" , "Jeff Sposetti" Date: Fri, 29 May 2015 11:53:36 -0000 Message-ID: <20150529115336.8660.41106@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Andrew Onischuk" X-ReviewGroup: Ambari X-ReviewRequest-URL: https://reviews.apache.org/r/34804/ X-Sender: "Andrew Onischuk" References: <20150529113002.8664.82817@reviews.apache.org> In-Reply-To: <20150529113002.8664.82817@reviews.apache.org> Reply-To: "Andrew Onischuk" X-ReviewRequest-Repository: ambari --===============1888250300247286755== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34804/ ----------------------------------------------------------- (Updated May 29, 2015, 11:53 a.m.) Review request for Ambari and Dmitro Lisnichenko. Bugs: AMBARI-11529 https://issues.apache.org/jira/browse/AMBARI-11529 Repository: ambari Description ------- Due to profiling below, from 420 seconds of deploy json.load and json.loads take 40 seconds (which even doesn't include working with json in ambari-agent itself, only script): 204 0.007 0.000 20.182 0.099 __init__.py:243(load) 351 0.002 0.000 17.894 0.051 __init__.py:270(loads) This is because json module for Python 2.6 is extremety slow. There are benchmark for it on here: due to it simplejson is at least ~40 times faster for deserializing. Also another thing to do in scope of this jira is to replace inspect module usage with os._getframe() which doesn't use realpath function that intensively, taking away 10 more seconds. 30049/21438 1.712 0.000 10.873 0.001 posixpath.py:354(realpath) Diffs (updated) ----- LICENSE.txt e226388 NOTICE.txt 9e59af8 ambari-agent/conf/unix/install-helper.sh 7c18b35 ambari-agent/pom.xml 9a58537 ambari-agent/src/main/python/ambari_agent/ActionQueue.py e486f34 ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py bae5362 ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py 8abb7ae ambari-agent/src/main/python/ambari_agent/AmbariConfig.py d4e0c20 ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py d59e230 ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py 861b568 ambari-agent/src/main/python/ambari_agent/Controller.py 4582d07 ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py e291196 ambari-agent/src/main/python/ambari_agent/Heartbeat.py ffef323 ambari-agent/src/main/python/ambari_agent/LiveStatus.py c56f6f3 ambari-agent/src/main/python/ambari_agent/PythonExecutor.py abbbe85 ambari-agent/src/main/python/ambari_agent/alerts/metric_alert.py a413027 ambari-agent/src/main/python/ambari_agent/security.py dbb2192 ambari-agent/src/test/python/ambari_agent/TestActionQueue.py 7a834d8 ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 422cb8c ambari-agent/src/test/python/ambari_agent/TestController.py e00e737 ambari-agent/src/test/python/ambari_agent/TestSecurity.py 52ddbdb ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 918aed1 ambari-common/src/main/python/ambari_simplejson/README.txt PRE-CREATION ambari-common/src/main/python/ambari_simplejson/__init__.py PRE-CREATION ambari-common/src/main/python/ambari_simplejson/_speedups.so PRE-CREATION ambari-common/src/main/python/ambari_simplejson/decoder.py PRE-CREATION ambari-common/src/main/python/ambari_simplejson/encoder.py PRE-CREATION ambari-common/src/main/python/ambari_simplejson/scanner.py PRE-CREATION ambari-common/src/main/python/resource_management/core/logger.py c741327 ambari-common/src/main/python/resource_management/core/shell.py 71b6501 ambari-common/src/main/python/resource_management/libraries/functions/curl_krb_request.py a66a25a ambari-common/src/main/python/resource_management/libraries/functions/flume_agent_helper.py 94f96ca ambari-common/src/main/python/resource_management/libraries/functions/jmx.py b32f6aa ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py eab01ad ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py b6be4c2 ambari-common/src/main/python/resource_management/libraries/functions/security_commons.py 1e92f9d ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 9f145a8 ambari-common/src/main/python/resource_management/libraries/script/script.py 3ee1f87 ambari-server/conf/unix/install-helper.sh 108f4c2 ambari-server/pom.xml 4158d95 ambari-server/src/main/python/ambari_server/serverUpgrade.py 3cba2ea ambari-server/src/main/python/ambari_server/setupSecurity.py c860ecd ambari-server/src/main/python/upgradeHelper.py 82c76bc ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py b4cff05 ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py 5ba5b03 ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py ee95493 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py 7b20a79 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py 40b5694 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 864961e ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 229607d ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 38270e8 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py df93633 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 6e12dd0 ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py b5cd909 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 9028800 ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py 2846919 ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm_upgrade.py 3f87694 ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py 80749e6 ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanagers_summary.py 45c8ea6 ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/files/validateYarnComponentStatusWindows.py 073371a ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py 8e378b5 ambari-server/src/main/resources/custom_actions/scripts/install_packages.py efe801b ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 32fb57a ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 837bd66 ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py 35267db ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py 32efb5c Diff: https://reviews.apache.org/r/34804/diff/ Testing ------- mvn clean test Thanks, Andrew Onischuk --===============1888250300247286755==--