ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk" <aonis...@hortonworks.com>
Subject Re: Review Request 34804: Work with json module takes too long
Date Fri, 29 May 2015 11:53:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34804/
-----------------------------------------------------------

(Updated May 29, 2015, 11:53 a.m.)


Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-11529
    https://issues.apache.org/jira/browse/AMBARI-11529


Repository: ambari


Description
-------

Due to profiling below, from 420 seconds of deploy json.load and json.loads
take 40 seconds (which even doesn't include working with json in ambari-agent
itself, only script):

    
    
          204    0.007    0.000   20.182    0.099 __init__.py:243(load)
          351    0.002    0.000   17.894    0.051 __init__.py:270(loads)
    

This is because json module for Python 2.6 is extremety slow. There are
benchmark for it on here:  
<http://stackoverflow.com/questions/706101/python-json-decoding-performance>

due to it simplejson is at least ~40 times faster for deserializing.

Also another thing to do in scope of this jira is to replace inspect module
usage with os._getframe() which doesn't use realpath function that
intensively, taking away 10 more seconds.

    
    
    30049/21438    1.712    0.000   10.873    0.001 posixpath.py:354(realpath)


Diffs (updated)
-----

  LICENSE.txt e226388 
  NOTICE.txt 9e59af8 
  ambari-agent/conf/unix/install-helper.sh 7c18b35 
  ambari-agent/pom.xml 9a58537 
  ambari-agent/src/main/python/ambari_agent/ActionQueue.py e486f34 
  ambari-agent/src/main/python/ambari_agent/ActualConfigHandler.py bae5362 
  ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py 8abb7ae 
  ambari-agent/src/main/python/ambari_agent/AmbariConfig.py d4e0c20 
  ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py d59e230 
  ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py 861b568 
  ambari-agent/src/main/python/ambari_agent/Controller.py 4582d07 
  ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py e291196 
  ambari-agent/src/main/python/ambari_agent/Heartbeat.py ffef323 
  ambari-agent/src/main/python/ambari_agent/LiveStatus.py c56f6f3 
  ambari-agent/src/main/python/ambari_agent/PythonExecutor.py abbbe85 
  ambari-agent/src/main/python/ambari_agent/alerts/metric_alert.py a413027 
  ambari-agent/src/main/python/ambari_agent/security.py dbb2192 
  ambari-agent/src/test/python/ambari_agent/TestActionQueue.py 7a834d8 
  ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 422cb8c 
  ambari-agent/src/test/python/ambari_agent/TestController.py e00e737 
  ambari-agent/src/test/python/ambari_agent/TestSecurity.py 52ddbdb 
  ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 918aed1 
  ambari-common/src/main/python/ambari_simplejson/README.txt PRE-CREATION 
  ambari-common/src/main/python/ambari_simplejson/__init__.py PRE-CREATION 
  ambari-common/src/main/python/ambari_simplejson/_speedups.so PRE-CREATION 
  ambari-common/src/main/python/ambari_simplejson/decoder.py PRE-CREATION 
  ambari-common/src/main/python/ambari_simplejson/encoder.py PRE-CREATION 
  ambari-common/src/main/python/ambari_simplejson/scanner.py PRE-CREATION 
  ambari-common/src/main/python/resource_management/core/logger.py c741327 
  ambari-common/src/main/python/resource_management/core/shell.py 71b6501 
  ambari-common/src/main/python/resource_management/libraries/functions/curl_krb_request.py
a66a25a 
  ambari-common/src/main/python/resource_management/libraries/functions/flume_agent_helper.py
94f96ca 
  ambari-common/src/main/python/resource_management/libraries/functions/jmx.py b32f6aa 
  ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions.py
eab01ad 
  ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py
b6be4c2 
  ambari-common/src/main/python/resource_management/libraries/functions/security_commons.py
1e92f9d 
  ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py 9f145a8

  ambari-common/src/main/python/resource_management/libraries/script/script.py 3ee1f87 
  ambari-server/conf/unix/install-helper.sh 108f4c2 
  ambari-server/pom.xml 4158d95 
  ambari-server/src/main/python/ambari_server/serverUpgrade.py 3cba2ea 
  ambari-server/src/main/python/ambari_server/setupSecurity.py c860ecd 
  ambari-server/src/main/python/upgradeHelper.py 82c76bc 
  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py
b4cff05 
  ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py
5ba5b03 
  ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
ee95493 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py
7b20a79 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py
40b5694 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
864961e 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
229607d 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
38270e8 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py
df93633 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
6e12dd0 
  ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py
b5cd909 
  ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py
9028800 
  ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py
2846919 
  ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/storm_upgrade.py
3f87694 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py
80749e6 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanagers_summary.py
45c8ea6 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/files/validateYarnComponentStatusWindows.py
073371a 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py
8e378b5 
  ambari-server/src/main/resources/custom_actions/scripts/install_packages.py efe801b 
  ambari-server/src/main/resources/custom_actions/scripts/ru_execute_tasks.py 32fb57a 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 837bd66

  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py
35267db 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/repo_initialization.py
32efb5c 

Diff: https://reviews.apache.org/r/34804/diff/


Testing
-------

mvn clean test


Thanks,

Andrew Onischuk


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message