Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DB9C9200CC6 for ; Mon, 3 Jul 2017 21:56:47 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id D859E160BF9; Mon, 3 Jul 2017 19:56:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8B7B8160BD2 for ; Mon, 3 Jul 2017 21:56:45 +0200 (CEST) Received: (qmail 94542 invoked by uid 500); 3 Jul 2017 19:56:44 -0000 Mailing-List: contact commits-help@ariatosca.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ariatosca.incubator.apache.org Delivered-To: mailing list commits@ariatosca.incubator.apache.org Received: (qmail 94522 invoked by uid 99); 3 Jul 2017 19:56:43 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2017 19:56:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 7166B1AFC7B for ; Mon, 3 Jul 2017 19:56:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.222 X-Spam-Level: X-Spam-Status: No, score=-4.222 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id FH4HCRem3zCx for ; Mon, 3 Jul 2017 19:56:28 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 29A0D60DA9 for ; Mon, 3 Jul 2017 19:56:23 +0000 (UTC) Received: (qmail 93384 invoked by uid 99); 3 Jul 2017 19:56:22 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2017 19:56:22 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 48564EEE19; Mon, 3 Jul 2017 19:56:19 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: emblemparade@apache.org To: commits@ariatosca.incubator.apache.org Date: Mon, 03 Jul 2017 19:56:40 -0000 Message-Id: <44591d9d7297426895c133e70e6d2661@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [23/31] incubator-ariatosca git commit: ARIA-286 Sphinx documentation for code and CLI archived-at: Mon, 03 Jul 2017 19:56:48 -0000 http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/presentation/fields.py ---------------------------------------------------------------------- diff --git a/aria/parser/presentation/fields.py b/aria/parser/presentation/fields.py index 7f85723..959bad1 100644 --- a/aria/parser/presentation/fields.py +++ b/aria/parser/presentation/fields.py @@ -39,19 +39,16 @@ def has_fields(cls): """ Class decorator for validated field support. - 1. Adds a :code:`FIELDS` class property that is a dict of all the fields. - Will inherit and merge :code:`FIELDS` properties from base classes if - they have them. + 1. Adds a ``FIELDS`` class property that is a dict of all the fields. Will inherit and merge + ``FIELDS`` properties from base classes if they have them. - 2. Generates automatic :code:`@property` implementations for the fields - with the help of a set of special function decorators. + 2. Generates automatic ``@property`` implementations for the fields with the help of a set of + special function decorators. - The class also works with the Python dict protocol, so that - fields can be accessed via dict semantics. The functionality is - identical to that of using attribute access. + The class also works with the Python dict protocol, so that fields can be accessed via dict + semantics. The functionality is identical to that of using attribute access. - The class will also gain two utility methods, :code:`_iter_field_names` - and :code:`_iter_fields`. + The class will also gain two utility methods, ``_iter_field_names`` and ``_iter_fields``. """ # Make sure we have FIELDS @@ -267,8 +264,8 @@ def field_getter(getter_func): """ Method decorator for overriding the getter function of a field. - The signature of the getter function must be: :code:`f(field, presentation, context)`. - The default getter can be accessed as :code:`field.default_get(presentation, context)`. + The signature of the getter function must be: ``f(field, presentation, context)``. + The default getter can be accessed as ```field.default_get(presentation, context)``. The function must already be decorated with a field decorator. """ @@ -286,8 +283,8 @@ def field_setter(setter_func): """ Method decorator for overriding the setter function of a field. - The signature of the setter function must be: :code:`f(field, presentation, context, value)`. - The default setter can be accessed as :code:`field.default_set(presentation, context, value)`. + The signature of the setter function must be: ``f(field, presentation, context, value)``. + The default setter can be accessed as ``field.default_set(presentation, context, value)``. The function must already be decorated with a field decorator. """ @@ -305,8 +302,8 @@ def field_validator(validator_fn): """ Method decorator for overriding the validator function of a field. - The signature of the validator function must be: :code:f(field, presentation, context)`. - The default validator can be accessed as :code:`field.default_validate(presentation, context)`. + The signature of the validator function must be: ``f(field, presentation, context)``. + The default validator can be accessed as ``field.default_validate(presentation, context)``. The function must already be decorated with a field decorator. """ @@ -373,7 +370,7 @@ def has_fields_contains(self, key): class Field(object): """ - Field handler used by :code:`@has_fields` decorator. + Field handler used by ``@has_fields`` decorator. """ def __init__(self, field_variant, func, cls=None, default=None, allowed=None, required=False): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/presentation/null.py ---------------------------------------------------------------------- diff --git a/aria/parser/presentation/null.py b/aria/parser/presentation/null.py index a018df4..287d2ba 100644 --- a/aria/parser/presentation/null.py +++ b/aria/parser/presentation/null.py @@ -33,7 +33,7 @@ NULL = Null() def none_to_null(value): """ - Convert :code:`None` to :code:`NULL`, recursively. + Convert ``None`` to ``NULL``, recursively. """ if value is None: @@ -51,7 +51,7 @@ def none_to_null(value): def null_to_none(value): """ - Convert :code:`NULL` to :code:`None`, recursively. + Convert ``NULL`` to ``None``, recursively. """ if value is NULL: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/presentation/presentation.py ---------------------------------------------------------------------- diff --git a/aria/parser/presentation/presentation.py b/aria/parser/presentation/presentation.py index 644d880..7292562 100644 --- a/aria/parser/presentation/presentation.py +++ b/aria/parser/presentation/presentation.py @@ -52,18 +52,17 @@ class PresentationBase(HasCachedMethods): def _validate(self, context): """ - Validates the presentation while reporting errors in the validation context but - *not* raising exceptions. + Validates the presentation while reporting errors in the validation context but *not* + raising exceptions. - The base class does not thing, but subclasses may override this for specialized - validation. + The base class does not thing, but subclasses may override this for specialized validation. """ @property def _fullname(self): """ - Always returns a usable full name of the presentation, whether it itself is named, - or recursing to its container, and finally defaulting to the class name. + Always returns a usable full name of the presentation, whether it itself is named, or + recursing to its container, and finally defaulting to the class name. """ if self._name is not None: @@ -75,10 +74,10 @@ class PresentationBase(HasCachedMethods): @property def _locator(self): """ - Attempts to return the most relevant locator, whether we have one, or recursing - to our container. + Attempts to return the most relevant locator, whether we have one, or recursing to our + container. - :rtype: :class:`aria.reading.Locator` + :rtype: :class:`aria.parser.reading.Locator` """ return get_locator(self._raw, self._container) @@ -98,8 +97,8 @@ class PresentationBase(HasCachedMethods): def _get_from_dict(self, *names): """ - Gets attributes recursively, except for the last name which is used - to get a value from the last dict. + Gets attributes recursively, except for the last name which is used to get a value from the + last dict. """ if names: @@ -110,10 +109,10 @@ class PresentationBase(HasCachedMethods): def _get_child_locator(self, *names): """ - Attempts to return the locator of one our children. Will default to our locator - if not found. + Attempts to return the locator of one our children. Will default to our locator if not + found. - :rtype: :class:`aria.reading.Locator` + :rtype: :class:`aria.parser.reading.Locator` """ if hasattr(self._raw, '_locator'): @@ -126,9 +125,8 @@ class PresentationBase(HasCachedMethods): """ Emits a colorized representation. - The base class will emit a sensible default representation of the fields, - (by calling :code:`_dump_content`), but subclasses may override this for specialized - dumping. + The base class will emit a sensible default representation of the fields, (by calling + ``_dump_content``), but subclasses may override this for specialized dumping. """ if self._name: @@ -142,8 +140,8 @@ class PresentationBase(HasCachedMethods): """ Emits a colorized representation of the contents. - The base class will call :code:`_dump_field` on all the fields, but subclasses may - override this for specialized dumping. + The base class will call ``_dump_field`` on all the fields, but subclasses may override + this for specialized dumping. """ if field_names: @@ -159,8 +157,8 @@ class PresentationBase(HasCachedMethods): """ Emits a colorized representation of the field. - According to the field type, this may trigger nested recursion. The nested - types will delegate to their :code:`_dump` methods. + According to the field type, this may trigger nested recursion. The nested types will + delegate to their ``_dump`` methods. """ field = self.FIELDS[field_name] # pylint: disable=no-member @@ -179,16 +177,14 @@ class PresentationBase(HasCachedMethods): class Presentation(PresentationBase): """ - Base class for ARIA presentations. A presentation is a Pythonic wrapper around - agnostic raw data, adding the ability to read and modify the data with proper - validation. + Base class for ARIA presentations. A presentation is a Pythonic wrapper around agnostic raw + data, adding the ability to read and modify the data with proper validation. - ARIA presentation classes will often be decorated with @has_fields, as that - mechanism automates a lot of field-specific validation. However, that is not a - requirement. + ARIA presentation classes will often be decorated with :func:`has_fields`, as that mechanism + automates a lot of field-specific validation. However, that is not a requirement. - Make sure that your utility property and method names begin with a "_", because - those names without a "_" prefix are normally reserved for fields. + Make sure that your utility property and method names begin with a ``_``, because those names + without a ``_`` prefix are normally reserved for fields. """ def _validate(self, context): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/presentation/source.py ---------------------------------------------------------------------- diff --git a/aria/parser/presentation/source.py b/aria/parser/presentation/source.py index 7198b07..4bfb8e1 100644 --- a/aria/parser/presentation/source.py +++ b/aria/parser/presentation/source.py @@ -32,7 +32,7 @@ class PresenterSource(object): class DefaultPresenterSource(PresenterSource): """ - The default ARIA presenter source supports TOSCA Simple Profile. + The default ARIA presenter source. """ def __init__(self, classes=None): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/presentation/utils.py ---------------------------------------------------------------------- diff --git a/aria/parser/presentation/utils.py b/aria/parser/presentation/utils.py index fbe971b..f0fd390 100644 --- a/aria/parser/presentation/utils.py +++ b/aria/parser/presentation/utils.py @@ -25,7 +25,7 @@ def get_locator(*values): """ Gets the first available locator. - :rtype: :class:`aria.reading.Locator` + :rtype: :class:`aria.parser.reading.Locator` """ for v in values: @@ -53,7 +53,7 @@ def validate_primitive(value, cls, coerce=False): Checks if the value is of the primitive type, optionally attempting to coerce it if it is not. - Raises a :code:`ValueError` if it isn't or if coercion failed. + :raises ValueError: if not a primitive type or if coercion failed. """ if (cls is not None) and (value is not None) and (value is not NULL): @@ -112,14 +112,14 @@ def validate_known_fields(context, presentation): def get_parent_presentation(context, presentation, *types_dict_names): """ - Returns the parent presentation according to the :code:`derived_from` field, or None if invalid. + Returns the parent presentation according to the ``derived_from`` field, or ``None`` if invalid. Checks that we do not derive from ourselves and that we do not cause a circular hierarchy. The arguments from the third onwards are used to locate a nested field under - :code:`service_template` under the root presenter. The first of these can optionally - be a function, in which case it will be called to convert type names. This can be used - to support shorthand type names, aliases, etc. + ``service_template`` under the root presenter. The first of these can optionally be a function, + in which case it will be called to convert type names. This can be used to support shorthand + type names, aliases, etc. """ type_name = presentation.derived_from http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/reading/__init__.py ---------------------------------------------------------------------- diff --git a/aria/parser/reading/__init__.py b/aria/parser/reading/__init__.py index b5c0709..065ca56 100644 --- a/aria/parser/reading/__init__.py +++ b/aria/parser/reading/__init__.py @@ -10,6 +10,29 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Reading package. + +.. autosummary:: + :nosignatures: + + aria.parser.reading.ReadingContext + ReaderException + ReaderNotFoundError + ReaderSyntaxError + AlreadyReadException + JinjaReader + JsonReader + Locator + deepcopy_with_locators + copy_locators + RawReader + Reader + ReaderSource + DefaultReaderSource + YamlReader +""" + from .raw import RawReader from .reader import Reader from .yaml import YamlReader http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/reading/context.py ---------------------------------------------------------------------- diff --git a/aria/parser/reading/context.py b/aria/parser/reading/context.py index 02d4de9..233e407 100644 --- a/aria/parser/reading/context.py +++ b/aria/parser/reading/context.py @@ -16,10 +16,12 @@ from .source import DefaultReaderSource class ReadingContext(object): """ - Properties: + Reading context. - * :code:`reader_source`: For finding reader instances - * :code:`reader`: Overrides :code:`reader_source` with a specific class + :ivar reader_source: for finding reader instances + :vartype reader_source: ReaderSource + :ivar reader: overrides ``reader_source`` with a specific class + :vartype reader: type """ def __init__(self): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/reading/locator.py ---------------------------------------------------------------------- diff --git a/aria/parser/reading/locator.py b/aria/parser/reading/locator.py index 4142ee7..965164d 100644 --- a/aria/parser/reading/locator.py +++ b/aria/parser/reading/locator.py @@ -124,7 +124,7 @@ class Locator(object): def deepcopy_with_locators(value): """ - Like :code:`deepcopy`, but also copies over locators. + Like :func:`deepcopy`, but also copies over locators. """ res = deepcopy(value) @@ -134,7 +134,7 @@ def deepcopy_with_locators(value): def copy_locators(target, source): """ - Copies over :code:`_locator` for all elements, recursively. + Copies over ``_locator`` for all elements, recursively. Assumes that target and source have exactly the same list/dict structure. """ http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/reading/reader.py ---------------------------------------------------------------------- diff --git a/aria/parser/reading/reader.py b/aria/parser/reading/reader.py index fe64697..1a29f11 100644 --- a/aria/parser/reading/reader.py +++ b/aria/parser/reading/reader.py @@ -18,7 +18,7 @@ class Reader(object): """ Base class for ARIA readers. - Readers provide agnostic raw data by consuming :class:`aria.loading.Loader` instances. + Readers provide agnostic raw data by consuming :class:`aria.parser.loading.Loader` instances. """ def __init__(self, context, location, loader): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/specification.py ---------------------------------------------------------------------- diff --git a/aria/parser/specification.py b/aria/parser/specification.py index 714bed1..4f452b8 100644 --- a/aria/parser/specification.py +++ b/aria/parser/specification.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Utilities for cross-referencing code with specification documents. +""" + import re from ..extension import parser http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/validation/__init__.py ---------------------------------------------------------------------- diff --git a/aria/parser/validation/__init__.py b/aria/parser/validation/__init__.py index fead43b..21632ba 100644 --- a/aria/parser/validation/__init__.py +++ b/aria/parser/validation/__init__.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Validation package. +""" + from .issue import Issue from .context import ValidationContext http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/parser/validation/context.py ---------------------------------------------------------------------- diff --git a/aria/parser/validation/context.py b/aria/parser/validation/context.py index d81f1cd..ef641bd 100644 --- a/aria/parser/validation/context.py +++ b/aria/parser/validation/context.py @@ -23,13 +23,16 @@ from ...utils.formatting import as_raw class ValidationContext(object): """ - Properties: + Validation context. - * :code:`allow_unknown_fields`: When False (the default) will report an issue - if an unknown field is used - * :code:`allow_primitive_coersion`: When False (the default) will not attempt to - coerce primitive field types - * :code:`max_level`: Maximum validation level to report (default is all) + :ivar allow_unknown_fields: when ``False`` (the default) will report an issue if an unknown + field is used + :vartype allow_unknown_fields: bool + :ivar allow_primitive_coersion`: when ``False`` (the default) will not attempt to coerce + primitive field types + :vartype allow_primitive_coersion: bool + :ivar max_level: maximum validation level to report (default is all) + :vartype max_level: int """ def __init__(self): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/storage/__init__.py ---------------------------------------------------------------------- diff --git a/aria/storage/__init__.py b/aria/storage/__init__.py index bd7c8c1..a553ca7 100644 --- a/aria/storage/__init__.py +++ b/aria/storage/__init__.py @@ -14,29 +14,9 @@ # limitations under the License. """ -ARIA's storage Sub-Package -Path: aria.storage - -Storage package is a generic abstraction over different storage types. -We define this abstraction with the following components: - -1. storage: simple mapi to use -2. driver: implementation of the database client mapi. -3. model: defines the structure of the table/document. -4. field: defines a field/item in the model. - -API: - * application_storage_factory - function, default ARIA storage factory. - * Storage - class, simple storage mapi. - * models - module, default ARIA standard models. - * structures - module, default ARIA structures - holds the base model, - and different fields types. - * Model - class, abstract model implementation. - * Field - class, base field implementation. - * IterField - class, base iterable field implementation. - * drivers - module, a pool of ARIA standard drivers. - * StorageDriver - class, abstract model implementation. +Storage package. """ + from .core import ( Storage, ModelStorage, http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/storage/api.py ---------------------------------------------------------------------- diff --git a/aria/storage/api.py b/aria/storage/api.py index 3304721..a337743 100644 --- a/aria/storage/api.py +++ b/aria/storage/api.py @@ -12,15 +12,17 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + """ -General storage API +Storage APIs. """ + import threading class StorageAPI(object): """ - General storage Base API + Base class for storage APIs. """ def create(self, **kwargs): """ @@ -33,15 +35,12 @@ class StorageAPI(object): class ModelAPI(StorageAPI): """ - A Base object for the model. + Base class for model APIs ("MAPI"). """ def __init__(self, model_cls, name=None, **kwargs): """ - Base model API - - :param model_cls: the representing class of the model - :param str name: the name of the model - :param kwargs: + :param model_cls: representing class of the model + :param name: name of the model """ super(ModelAPI, self).__init__(**kwargs) self._model_cls = model_cls @@ -59,46 +58,42 @@ class ModelAPI(StorageAPI): @property def name(self): """ - The name of the class - :return: name of the class + Name of the class. + + :type: :obj:`basestring` """ return self._name @property def model_cls(self): """ - The class represting the model - :return: + Class representing the model + + :type: :obj:`Type` """ return self._model_cls def get(self, entry_id, filters=None, **kwargs): """ - Get entry from storage. + Gets a model from storage. :param entry_id: - :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract get method') def put(self, entry, **kwargs): """ - Store entry in storage + Puts a model in storage. :param entry: - :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract store method') def delete(self, entry_id, **kwargs): """ - Delete entry from storage. + Deletes a model from storage. :param entry_id: - :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract delete method') @@ -107,32 +102,27 @@ class ModelAPI(StorageAPI): def iter(self, **kwargs): """ - Iter over the entries in storage. - - :param kwargs: - :return: + Iterate over all models in storage. """ raise NotImplementedError('Subclass must implement abstract iter method') def update(self, entry, **kwargs): """ - Update entry in storage. + Update a model in storage. :param entry: :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract update method') class ResourceAPI(StorageAPI): """ - A Base object for the resource. + Base class for resource APIs ("RAPI"). """ def __init__(self, name, **kwargs): """ - Base resource API - :param str name: the resource type + :param name: resource type """ super(ResourceAPI, self).__init__(**kwargs) self._name = name @@ -140,63 +130,57 @@ class ResourceAPI(StorageAPI): @property def name(self): """ - The name of the resource - :return: + Name of resource. + + :type: :obj:`basestring` """ return self._name def read(self, entry_id, path, **kwargs): """ - Get a bytesteam from the storage. + Get a bytesteam for a resource from storage. :param entry_id: :param path: - :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract read method') def delete(self, entry_id, path, **kwargs): """ - Delete a resource from the storage. + Delete a resource from storage. :param entry_id: :param path: - :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract delete method') def download(self, entry_id, destination, path=None, **kwargs): """ - Download a resource from the storage. + Download a resource from storage. :param entry_id: :param destination: :param path: - :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract download method') def upload(self, entry_id, source, path=None, **kwargs): """ - Upload a resource to the storage. + Upload a resource to storage. :param entry_id: :param source: :param path: - :param kwargs: - :return: """ raise NotImplementedError('Subclass must implement abstract upload method') def generate_lower_name(model_cls): """ - Generates the name of the class from the class object. e.g. SomeClass -> some_class - :param model_cls: the class to evaluate. - :return: lower name + Generates the name of the class from the class object, e.g. ``SomeClass`` -> ``some_class`` + + :param model_cls: class to evaluate + :return: lowercase name :rtype: basestring """ return getattr(model_cls, '__mapiname__', model_cls.__tablename__) http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/storage/collection_instrumentation.py ---------------------------------------------------------------------- diff --git a/aria/storage/collection_instrumentation.py b/aria/storage/collection_instrumentation.py index 454f97a..c90cb18 100644 --- a/aria/storage/collection_instrumentation.py +++ b/aria/storage/collection_instrumentation.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Utilities for instrumenting collections of models in storage. +""" + from . import exceptions @@ -42,17 +46,15 @@ class _InstrumentedCollection(object): Instantiates the object from existing seq. :param seq: the original sequence to load from - :return: """ raise NotImplementedError def _set(self, key, value): """ - set the changes for the current object (not in the db) + Sets the changes for the current object (not in the database). :param key: :param value: - :return: """ raise NotImplementedError @@ -61,10 +63,10 @@ class _InstrumentedCollection(object): def _instrument(self, key, value): """ - Instruments any collection to track changes (and ease of access) + Instruments any collection to track changes (and ease of access). + :param key: :param value: - :return: """ if isinstance(value, _InstrumentedCollection): return value @@ -79,9 +81,9 @@ class _InstrumentedCollection(object): def _raw_value(self, value): """ - Get the raw value. + Gets the raw value. + :param value: - :return: """ if isinstance(value, self._field_cls): return value.value @@ -89,10 +91,10 @@ class _InstrumentedCollection(object): def _encapsulate_value(self, key, value): """ - Create a new item cls if needed. + Creates a new item class if needed. + :param key: :param value: - :return: """ if isinstance(value, self._field_cls): return value @@ -101,10 +103,10 @@ class _InstrumentedCollection(object): def __setitem__(self, key, value): """ - Update the values in both the local and the db locations. + Updates the values in both the local and the database locations. + :param key: :param value: - :return: """ self._set(key, value) if self._is_top_level: @@ -119,11 +121,11 @@ class _InstrumentedCollection(object): def _set_field(self, collection, key, value): """ - enables updating the current change in the ancestors - :param collection: the collection to change - :param key: the key for the specific field - :param value: the new value - :return: + Enables updating the current change in the ancestors. + + :param collection: collection to change + :param key: key for the specific field + :param value: new value """ if isinstance(value, _InstrumentedCollection): value = value._raw @@ -209,9 +211,10 @@ class _InstrumentedModel(_WrappedBase): def __init__(self, mapi, *args, **kwargs): """ - The original model - :param wrapped: the model to be instrumented - :param mapi: the mapi for that model + The original model. + + :param wrapped: model to be instrumented + :param mapi: MAPI for the wrapped model """ super(_InstrumentedModel, self).__init__(*args, **kwargs) self._mapi = mapi @@ -261,11 +264,10 @@ class _WrappedModel(_WrappedBase): def __init__(self, instrumentation_kwargs, *args, **kwargs): """ - - :param instrumented_cls: The class to be instrumented - :param instrumentation_cls: the instrumentation cls - :param wrapped: the currently wrapped instance - :param kwargs: and kwargs to the passed to the instrumented class. + :param instrumented_cls: class to be instrumented + :param instrumentation_cls: instrumentation cls + :param wrapped: currently wrapped instance + :param kwargs: passed to the instrumented class """ super(_WrappedModel, self).__init__(*args, **kwargs) self._kwargs = instrumentation_kwargs http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/storage/core.py ---------------------------------------------------------------------- diff --git a/aria/storage/core.py b/aria/storage/core.py index 06c29e8..74b1147 100644 --- a/aria/storage/core.py +++ b/aria/storage/core.py @@ -14,29 +14,9 @@ # limitations under the License. """ -ARIA's storage Sub-Package -Path: aria.storage - -Storage package is a generic abstraction over different storage types. -We define this abstraction with the following components: - -1. storage: simple mapi to use -2. driver: implementation of the database client mapi. -3. model: defines the structure of the table/document. -4. field: defines a field/item in the model. - -API: - * application_storage_factory - function, default ARIA storage factory. - * Storage - class, simple storage mapi. - * models - module, default ARIA standard models. - * structures - module, default ARIA structures - holds the base model, - and different fields types. - * Model - class, abstract model implementation. - * Field - class, base field implementation. - * IterField - class, base iterable field implementation. - * drivers - module, a pool of ARIA standard drivers. - * StorageDriver - class, abstract model implementation. +Storage API management. """ + import copy from contextlib import contextmanager @@ -52,7 +32,7 @@ __all__ = ( class Storage(LoggerMixin): """ - Represents the storage + Base class for storage managers. """ def __init__(self, api_cls, @@ -62,13 +42,12 @@ class Storage(LoggerMixin): initiator_kwargs=None, **kwargs): """ - - :param api_cls: API cls for each model. + :param api_cls: API class for each entry :param api_kwargs: - :param items: the items to register - :param initiator: a func which initializes the storage before the first use. - This function should return a dict, this dict would be passed in addition to the api kwargs. - This enables the creation of any unpickable objects across process. + :param items: items to register + :param initiator: function which initializes the storage before the first use; this function + should return a dict, this dict would be passed in addition to the API kwargs; this enables + the creation of non-serializable objects :param initiator_kwargs: :param kwargs: """ @@ -112,22 +91,22 @@ class Storage(LoggerMixin): def register(self, entry): """ - Register the entry to the storage - :param name: - :return: + Register an API. + + :param entry: """ raise NotImplementedError('Subclass must implement abstract register method') class ResourceStorage(Storage): """ - Represents resource storage. + Manages storage resource APIs ("RAPIs"). """ def register(self, name): """ - Register the resource type to resource storage. - :param name: - :return: + Register a storage resource API ("RAPI"). + + :param name: name """ self.registered[name] = self.api(name=name, **self._all_api_kwargs) self.registered[name].create() @@ -136,7 +115,7 @@ class ResourceStorage(Storage): class ModelStorage(Storage): """ - Represents model storage. + Manages storage model APIs ("MAPIs"). """ def __init__(self, *args, **kwargs): if kwargs.get('initiator', None) is None: @@ -145,9 +124,9 @@ class ModelStorage(Storage): def register(self, model_cls): """ - Register the model into the model storage. - :param model_cls: the model to register. - :return: + Register a storage model API ("MAPI"). + + :param model_cls: model API to register """ model_name = model_cls.__modelname__ if model_name in self.registered: @@ -162,8 +141,7 @@ class ModelStorage(Storage): def drop(self): """ - Drop all the tables from the model. - :return: + Drop all the tables. """ for mapi in self.registered.values(): mapi.drop() http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/storage/exceptions.py ---------------------------------------------------------------------- diff --git a/aria/storage/exceptions.py b/aria/storage/exceptions.py index 3f0ecec..c538876 100644 --- a/aria/storage/exceptions.py +++ b/aria/storage/exceptions.py @@ -12,9 +12,11 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + """ -Storage based exceptions +Storage exceptions. """ + from .. import exceptions http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/storage/filesystem_rapi.py ---------------------------------------------------------------------- diff --git a/aria/storage/filesystem_rapi.py b/aria/storage/filesystem_rapi.py index 3ddc520..b425fa2 100644 --- a/aria/storage/filesystem_rapi.py +++ b/aria/storage/filesystem_rapi.py @@ -12,9 +12,11 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + """ -File system based RAPI +File system implementation of the storage resource API ("RAPI"). """ + import os import shutil from multiprocessing import RLock @@ -30,13 +32,12 @@ from aria.storage import ( class FileSystemResourceAPI(api.ResourceAPI): """ - File system resource storage. + File system implementation of the storage resource API ("RAPI"). """ def __init__(self, directory, **kwargs): """ - File system implementation for storage api. - :param str directory: root dir for storage. + :param directory: root dir for storage """ super(FileSystemResourceAPI, self).__init__(**kwargs) self.directory = directory @@ -47,8 +48,7 @@ class FileSystemResourceAPI(api.ResourceAPI): @contextmanager def connect(self): """ - Established a connection and destroys it after use. - :return: + Establishes a connection and destroys it after use. """ try: self._establish_connection() @@ -60,16 +60,13 @@ class FileSystemResourceAPI(api.ResourceAPI): def _establish_connection(self): """ - Establish a conenction. used in the 'connect' contextmanager. - :return: + Establishes a connection. Used in the ``connect`` context manager. """ self._lock.acquire() - def _destroy_connection(self): """ - Destroy a connection. used in the 'connect' contextmanager. - :return: + Destroys a connection. Used in the ``connect`` context manager. """ self._lock.release() @@ -79,9 +76,9 @@ class FileSystemResourceAPI(api.ResourceAPI): def create(self, **kwargs): """ - Create directory in storage by path. - tries to create the root directory as well. - :param str name: path of file in storage. + Creates a directory in by path. Tries to create the root directory as well. + + :param name: path of directory """ try: os.makedirs(self.directory) @@ -94,11 +91,11 @@ class FileSystemResourceAPI(api.ResourceAPI): def read(self, entry_id, path, **_): """ - Retrieve the content of a file system storage resource. + Retrieves the contents of a file. - :param str entry_id: the id of the entry. - :param str path: a path to the specific resource to read. - :return: the content of the file. + :param entry_id: entry ID + :param path: path to resource + :return: contents of the file :rtype: bytes """ resource_relative_path = os.path.join(self.name, entry_id, path or '') @@ -118,11 +115,11 @@ class FileSystemResourceAPI(api.ResourceAPI): def download(self, entry_id, destination, path=None, **_): """ - Download a specific file or dir from the file system resource storage. + Downloads a file or directory. - :param str entry_id: the id of the entry. - :param str destination: the destination to download to - :param str path: the path to download relative to the root of the entry (otherwise all). + :param entry_id: entry ID + :param destination: download destination + :param path: path to download relative to the root of the entry (otherwise all) """ resource_relative_path = os.path.join(self.name, entry_id, path or '') resource = os.path.join(self.directory, resource_relative_path) @@ -136,10 +133,10 @@ class FileSystemResourceAPI(api.ResourceAPI): def upload(self, entry_id, source, path=None, **_): """ - Uploads a specific file or dir to the file system resource storage. + Uploads a file or directory. - :param str entry_id: the id of the entry. - :param source: the source of the files to upload. + :param entry_id: entry ID + :param source: source of the files to upload :param path: the destination of the file/s relative to the entry root dir. """ resource_directory = os.path.join(self.directory, self.name, entry_id) @@ -153,10 +150,10 @@ class FileSystemResourceAPI(api.ResourceAPI): def delete(self, entry_id, path=None, **_): """ - Deletes a file system storage resource. + Deletes a file or directory. - :param str entry_id: the id of the entry. - :param str path: a path to delete relative to the root of the entry (otherwise all). + :param entry_id: entry ID + :param path: path to delete relative to the root of the entry (otherwise all) """ destination = os.path.join(self.directory, self.name, entry_id, path or '') if os.path.exists(destination): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/storage/sql_mapi.py ---------------------------------------------------------------------- diff --git a/aria/storage/sql_mapi.py b/aria/storage/sql_mapi.py index bb6223a..975ada7 100644 --- a/aria/storage/sql_mapi.py +++ b/aria/storage/sql_mapi.py @@ -12,9 +12,11 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + """ -SQLAlchemy based MAPI +SQLAlchemy implementation of the storage model API ("MAPI"). """ + import os import platform @@ -42,7 +44,7 @@ _predicates = {'ge': '__ge__', class SQLAlchemyModelAPI(api.ModelAPI): """ - SQL based MAPI. + SQLAlchemy implementation of the storage model API ("MAPI"). """ def __init__(self, @@ -54,7 +56,8 @@ class SQLAlchemyModelAPI(api.ModelAPI): self._session = session def get(self, entry_id, include=None, **kwargs): - """Return a single result based on the model class and element ID + """ + Returns a single result based on the model class and element ID """ query = self._get_query(include, {'id': entry_id}) result = query.first() @@ -102,25 +105,28 @@ class SQLAlchemyModelAPI(api.ModelAPI): filters=None, sort=None, **kwargs): - """Return a (possibly empty) list of `model_class` results + """ + Returns a (possibly empty) list of ``model_class`` results. """ for result in self._get_query(include, filters, sort): yield self._instrument(result) def put(self, entry, **kwargs): - """Create a `model_class` instance from a serializable `model` object + """ + Creatse a ``model_class`` instance from a serializable ``model`` object. - :param entry: A dict with relevant kwargs, or an instance of a class - that has a `to_dict` method, and whose attributes match the columns - of `model_class` (might also my just an instance of `model_class`) - :return: An instance of `model_class` + :param entry: dict with relevant kwargs, or an instance of a class that has a ``to_dict`` + method, and whose attributes match the columns of ``model_class`` (might also be just an + instance of ``model_class``) + :return: an instance of ``model_class`` """ self._session.add(entry) self._safe_commit() return entry def delete(self, entry, **kwargs): - """Delete a single result based on the model class and element ID + """ + Deletes a single result based on the model class and element ID. """ self._load_relationships(entry) self._session.delete(entry) @@ -128,17 +134,19 @@ class SQLAlchemyModelAPI(api.ModelAPI): return entry def update(self, entry, **kwargs): - """Add `instance` to the DB session, and attempt to commit + """ + Adds ``instance`` to the database session, and attempts to commit. - :return: The updated instance + :return: updated instance """ return self.put(entry) def refresh(self, entry): - """Reload the instance with fresh information from the DB + """ + Reloads the instance with fresh information from the database. - :param entry: Instance to be re-loaded from the DB - :return: The refreshed instance + :param entry: instance to be re-loaded from the database + :return: refreshed instance """ self._session.refresh(entry) self._load_relationships(entry) @@ -160,14 +168,14 @@ class SQLAlchemyModelAPI(api.ModelAPI): def drop(self): """ - Drop the table from the storage. - :return: + Drops the table. """ self.model_cls.__table__.drop(self._engine) def _safe_commit(self): - """Try to commit changes in the session. Roll back if exception raised - Excepts SQLAlchemy errors and rollbacks if they're caught + """ + Try to commit changes in the session. Roll back if exception raised SQLAlchemy errors and + rolls back if they're caught. """ try: self._session.commit() @@ -179,11 +187,11 @@ class SQLAlchemyModelAPI(api.ModelAPI): raise exceptions.StorageError('SQL Storage error: {0}'.format(str(e))) def _get_base_query(self, include, joins): - """Create the initial query from the model class and included columns + """ + Create the initial query from the model class and included columns. - :param include: A (possibly empty) list of columns to include in - the query - :return: An SQLAlchemy AppenderQuery object + :param include: (possibly empty) list of columns to include in the query + :return: SQLAlchemy AppenderQuery object """ # If only some columns are included, query through the session object if include: @@ -199,9 +207,10 @@ class SQLAlchemyModelAPI(api.ModelAPI): @staticmethod def _get_joins(model_class, columns): - """Get a list of all the tables on which we need to join + """ + Gets a list of all the tables on which we need to join. - :param columns: A set of all attributes involved in the query + :param columns: set of all attributes involved in the query """ # Using a list instead of a set because order is important @@ -222,12 +231,13 @@ class SQLAlchemyModelAPI(api.ModelAPI): @staticmethod def _sort_query(query, sort=None): - """Add sorting clauses to the query + """ + Adds sorting clauses to the query. - :param query: Base SQL query - :param sort: An optional dictionary where keys are column names to - sort by, and values are the order (asc/desc) - :return: An SQLAlchemy AppenderQuery object + :param query: base SQL query + :param sort: optional dictionary where keys are column names to sort by, and values are + the order (asc/desc) + :return: SQLAlchemy AppenderQuery object """ if sort: for column, order in sort.items(): @@ -237,13 +247,13 @@ class SQLAlchemyModelAPI(api.ModelAPI): return query def _filter_query(self, query, filters): - """Add filter clauses to the query + """ + Adds filter clauses to the query. - :param query: Base SQL query - :param filters: An optional dictionary where keys are column names to - filter by, and values are values applicable for those columns (or lists - of such values) - :return: An SQLAlchemy AppenderQuery object + :param query: base SQL query + :param filters: optional dictionary where keys are column names to filter by, and values + are values applicable for those columns (or lists of such values) + :return: SQLAlchemy AppenderQuery object """ return self._add_value_filter(query, filters) @@ -264,17 +274,16 @@ class SQLAlchemyModelAPI(api.ModelAPI): include=None, filters=None, sort=None): - """Get an SQL query object based on the params passed - - :param model_class: SQL DB table class - :param include: An optional list of columns to include in the query - :param filters: An optional dictionary where keys are column names to - filter by, and values are values applicable for those columns (or lists - of such values) - :param sort: An optional dictionary where keys are column names to - sort by, and values are the order (asc/desc) - :return: A sorted and filtered query with only the relevant - columns + """ + Gets a SQL query object based on the params passed. + + :param model_class: SQL database table class + :param include: optional list of columns to include in the query + :param filters: optional dictionary where keys are column names to filter by, and values + are values applicable for those columns (or lists of such values) + :param sort: optional dictionary where keys are column names to sort by, and values are the + order (asc/desc) + :return: sorted and filtered query with only the relevant columns """ include, filters, sort, joins = self._get_joins_and_converted_columns( include, filters, sort @@ -305,9 +314,10 @@ class SQLAlchemyModelAPI(api.ModelAPI): include, filters, sort): - """Get a list of tables on which we need to join and the converted - `include`, `filters` and `sort` arguments (converted to actual SQLA - column/label objects instead of column names) + """ + Gets a list of tables on which we need to join and the converted ``include``, ``filters`` + and ```sort`` arguments (converted to actual SQLAlchemy column/label objects instead of + column names). """ include = include or [] filters = filters or dict() @@ -325,8 +335,9 @@ class SQLAlchemyModelAPI(api.ModelAPI): include, filters, sort): - """Go over the optional parameters (include, filters, sort), and - replace column names with actual SQLA column objects + """ + Gooes over the optional parameters (include, filters, sort), and replace column names with + actual SQLAlechmy column objects. """ include = [self._get_column(c) for c in include] filters = dict((self._get_column(c), filters[c]) for c in filters) @@ -335,9 +346,10 @@ class SQLAlchemyModelAPI(api.ModelAPI): return include, filters, sort def _get_column(self, column_name): - """Return the column on which an action (filtering, sorting, etc.) - would need to be performed. Can be either an attribute of the class, - or an association proxy linked to a relationship the class has + """ + Returns the column on which an action (filtering, sorting, etc.) would need to be performed. + Can be either an attribute of the class, or an association proxy linked to a relationship + in the class. """ column = getattr(self.model_cls, column_name) if column.is_attribute: @@ -352,15 +364,16 @@ class SQLAlchemyModelAPI(api.ModelAPI): @staticmethod def _paginate(query, pagination): - """Paginate the query by size and offset - - :param query: Current SQLAlchemy query object - :param pagination: An optional dict with size and offset keys - :return: A tuple with four elements: - - res ults: `size` items starting from `offset` - - the total count of items - - `size` [default: 0] - - `offset` [default: 0] + """ + Paginates the query by size and offset. + + :param query: current SQLAlchemy query object + :param pagination: optional dict with size and offset keys + :return: tuple with four elements: + * results: ``size`` items starting from ``offset`` + * the total count of items + * ``size`` [default: 0] + * ``offset`` [default: 0] """ if pagination: size = pagination.get('size', 0) @@ -374,8 +387,9 @@ class SQLAlchemyModelAPI(api.ModelAPI): @staticmethod def _load_relationships(instance): - """A helper method used to overcome a problem where the relationships - that rely on joins aren't being loaded automatically + """ + Helper method used to overcome a problem where the relationships that rely on joins aren't + being loaded automatically. """ for rel in instance.__mapper__.relationships: getattr(instance, rel.key) @@ -389,13 +403,15 @@ class SQLAlchemyModelAPI(api.ModelAPI): def init_storage(base_dir, filename='db.sqlite'): """ - A builtin ModelStorage initiator. - Creates a sqlalchemy engine and a session to be passed to the mapi. + Built-in ModelStorage initiator. + + Creates a SQLAlchemy engine and a session to be passed to the MAPI. + + ``initiator_kwargs`` must be passed to the ModelStorage which must hold the ``base_dir`` for the + location of the database file, and an option filename. This would create an SQLite database. - Initiator_kwargs must be passed to the ModelStorage which must hold the base_dir for the - location of the db file, and an option filename. This would create an sqlite db. - :param base_dir: the dir of the db - :param filename: the db file name. + :param base_dir: directory of the database + :param filename: database file name. :return: """ uri = 'sqlite:///{platform_char}{path}'.format( @@ -415,7 +431,7 @@ def init_storage(base_dir, filename='db.sqlite'): class ListResult(list): """ - a ListResult contains results about the requested items. + Contains results about the requested items. """ def __init__(self, metadata, *args, **qwargs): super(ListResult, self).__init__(*args, **qwargs) http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/__init__.py ---------------------------------------------------------------------- diff --git a/aria/utils/__init__.py b/aria/utils/__init__.py index ae1e83e..2a957a9 100644 --- a/aria/utils/__init__.py +++ b/aria/utils/__init__.py @@ -12,3 +12,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + +""" +General-purpose utilities package. +""" http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/archive.py ---------------------------------------------------------------------- diff --git a/aria/utils/archive.py b/aria/utils/archive.py index 63d9004..29efcb1 100644 --- a/aria/utils/archive.py +++ b/aria/utils/archive.py @@ -10,6 +10,9 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Archive utilities. +""" import os import tarfile http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/argparse.py ---------------------------------------------------------------------- diff --git a/aria/utils/argparse.py b/aria/utils/argparse.py index 365c148..a05a841 100644 --- a/aria/utils/argparse.py +++ b/aria/utils/argparse.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Enhancements to Python's ``argparse`` module. +""" + from __future__ import absolute_import # so we can import standard 'argparse' from argparse import ArgumentParser as BaseArgumentParser @@ -27,7 +31,7 @@ class ArgumentParser(BaseArgumentParser): def add_flag_argument(self, name, help_true=None, help_false=None, default=False): """ - Adds a flag argument as two arguments: :code:`--my-flag` and :code:`--no-my-flag`. + Adds a flag argument as two arguments: ``--my-flag`` and ``--no-my-flag``. """ dest = name.replace('-', '_') http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/caching.py ---------------------------------------------------------------------- diff --git a/aria/utils/caching.py b/aria/utils/caching.py index c9e475a..5f8cd88 100644 --- a/aria/utils/caching.py +++ b/aria/utils/caching.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Caching utilities. +""" + from __future__ import absolute_import # so we can import standard 'collections' and 'threading' from threading import Lock @@ -27,8 +31,8 @@ class cachedmethod(object): # pylint: disable=invalid-name The implementation is thread-safe. - Supports :code:`cache_info` to be compatible with Python 3's :code:`functools.lru_cache`. - Note that the statistics are combined for all instances of the class. + Supports ``cache_info`` to be compatible with Python 3's ``functools.lru_cache``. Note that the + statistics are combined for all instances of the class. Won't use the cache if not called when bound to an object, allowing you to override the cache. @@ -39,6 +43,7 @@ class cachedmethod(object): # pylint: disable=invalid-name ENABLED = True def __init__(self, func): + self.__doc__ = func.__doc__ self.func = func self.hits = 0 self.misses = 0 http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/collections.py ---------------------------------------------------------------------- diff --git a/aria/utils/collections.py b/aria/utils/collections.py index 1e732aa..ccc37a1 100644 --- a/aria/utils/collections.py +++ b/aria/utils/collections.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Additional collection classes and collection utilities. +""" + from __future__ import absolute_import # so we can import standard 'collections' from copy import deepcopy @@ -32,8 +36,8 @@ class FrozenList(list): """ An immutable list. - After initialization it will raise :class:`TypeError` exceptions if modification - is attempted. + After initialization it will raise :class:`~exceptions.TypeError` exceptions if modification is + attempted. Note that objects stored in the list may not be immutable. """ @@ -78,12 +82,13 @@ class FrozenList(list): EMPTY_READ_ONLY_LIST = FrozenList() + class FrozenDict(OrderedDict): """ An immutable ordered dict. - After initialization it will raise :class:`TypeError` exceptions if modification - is attempted. + After initialization it will raise :class:`~exceptions.TypeError` exceptions if modification is + attempted. Note that objects stored in the dict may not be immutable. """ @@ -109,9 +114,11 @@ class FrozenDict(OrderedDict): EMPTY_READ_ONLY_DICT = FrozenDict() + class StrictList(list): """ - A list that raises :class:`TypeError` exceptions when objects of the wrong type are inserted. + A list that raises :class:`~exceptions.TypeError` exceptions when objects of the wrong type are + inserted. """ def __init__(self, @@ -168,10 +175,11 @@ class StrictList(list): value = self._wrap(value) return super(StrictList, self).insert(index, value) + class StrictDict(OrderedDict): """ - An ordered dict that raises :class:`TypeError` exceptions - when keys or values of the wrong type are used. + An ordered dict that raises :class:`~exceptions.TypeError` exceptions when keys or values of the + wrong type are used. """ def __init__(self, @@ -211,6 +219,7 @@ class StrictDict(OrderedDict): value = self.wrapper_function(value) return super(StrictDict, self).__setitem__(key, value) + def merge(dict_a, dict_b, path=None, strict=False): """ Merges dicts, recursively. @@ -234,12 +243,14 @@ def merge(dict_a, dict_b, path=None, strict=False): dict_a[key] = value_b return dict_a + def is_removable(_container, _key, v): return (v is None) or ((isinstance(v, dict) or isinstance(v, list)) and (len(v) == 0)) + def prune(value, is_removable_function=is_removable): """ - Deletes :code:`None` and empty lists and dicts, recursively. + Deletes ``None`` and empty lists and dicts, recursively. """ if isinstance(value, list): @@ -258,10 +269,11 @@ def prune(value, is_removable_function=is_removable): return value -# TODO (left for tal): Move following two methods to some place parser specific +# TODO: Move following two methods to some place parser specific + def deepcopy_with_locators(value): """ - Like :code:`deepcopy`, but also copies over locators. + Like :func:`~copy.deepcopy`, but also copies over locators. """ res = deepcopy(value) @@ -271,7 +283,7 @@ def deepcopy_with_locators(value): def copy_locators(target, source): """ - Copies over :code:`_locator` for all elements, recursively. + Copies over ``_locator`` for all elements, recursively. Assumes that target and source have exactly the same list/dict structure. """ http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/console.py ---------------------------------------------------------------------- diff --git a/aria/utils/console.py b/aria/utils/console.py index 55d2529..642cbb1 100644 --- a/aria/utils/console.py +++ b/aria/utils/console.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Abstraction API above terminal color libraries. +""" + from clint.textui.core import STDOUT from clint.textui import puts as _puts from clint.textui.colored import ColoredString as _ColoredString http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/exceptions.py ---------------------------------------------------------------------- diff --git a/aria/utils/exceptions.py b/aria/utils/exceptions.py index b60cee4..5bb0e6d 100644 --- a/aria/utils/exceptions.py +++ b/aria/utils/exceptions.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Utilities for extracting and formatting Python exceptions. +""" + import sys import linecache import StringIO http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/file.py ---------------------------------------------------------------------- diff --git a/aria/utils/file.py b/aria/utils/file.py index 6d1aa16..75f2859 100644 --- a/aria/utils/file.py +++ b/aria/utils/file.py @@ -13,13 +13,19 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +File utilities. +""" + import errno import os import shutil def makedirs(path): - """An extension of os.makedirs that doesn't fail if the directory already exists""" + """ + Enhancement of :func:`os.makedirs` that doesn't fail if the directory already exists. + """ if os.path.isdir(path): return try: @@ -29,7 +35,6 @@ def makedirs(path): raise def remove_if_exists(path): - try: if os.path.isfile(path): os.remove(path) http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/formatting.py ---------------------------------------------------------------------- diff --git a/aria/utils/formatting.py b/aria/utils/formatting.py index b8d24cd..fa34b7d 100644 --- a/aria/utils/formatting.py +++ b/aria/utils/formatting.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +String formatting and string-based format utilities. +""" + import json from types import MethodType @@ -42,8 +46,7 @@ yaml.representer.RoundTripRepresenter.add_representer( class JsonAsRawEncoder(json.JSONEncoder): """ - A :class:`JSONEncoder` that will use the :code:`as_raw` property of objects - if available. + A :class:`JSONEncoder` that will use the ``as_raw`` property of objects if available. """ def raw_encoder_default(self, obj): try: @@ -61,8 +64,7 @@ class JsonAsRawEncoder(json.JSONEncoder): class YamlAsRawDumper(yaml.dumper.RoundTripDumper): # pylint: disable=too-many-ancestors """ - A :class:`RoundTripDumper` that will use the :code:`as_raw` property of objects - if available. + A :class:`RoundTripDumper` that will use the ``as_raw`` property of objects if available. """ def represent_data(self, data): @@ -101,8 +103,8 @@ def decode_dict(data): def safe_str(value): """ - Like :code:`str` coercion, but makes sure that Unicode strings are properly - encoded, and will never return None. + Like :class:`str` coercion, but makes sure that Unicode strings are properly encoded, and will + never return ``None``. """ try: @@ -113,7 +115,7 @@ def safe_str(value): def safe_repr(value): """ - Like :code:`repr`, but calls :code:`as_raw` and :code:`as_agnostic` first. + Like :func:`repr`, but calls :func:`as_raw` and :func:`as_agnostic` first. """ return repr(as_agnostic(as_raw(value))) @@ -143,7 +145,7 @@ def pluralize(noun): def as_raw(value): """ - Converts values using their :code:`as_raw` property, if it exists, recursively. + Converts values using their ``as_raw`` property, if it exists, recursively. """ if hasattr(value, 'as_raw'): @@ -164,7 +166,7 @@ def as_raw(value): def as_raw_list(value): """ - Assuming value is a list, converts its values using :code:`as_raw`. + Assuming value is a list, converts its values using :func:`as_raw`. """ if value is None: @@ -176,8 +178,7 @@ def as_raw_list(value): def as_raw_dict(value): """ - Assuming value is a dict, converts its values using :code:`as_raw`. - The keys are left as is. + Assuming value is a dict, converts its values using :func:`as_raw`. The keys are left as is. """ if value is None: @@ -188,8 +189,8 @@ def as_raw_dict(value): def as_agnostic(value): """ - Converts subclasses of list and dict to standard lists and dicts, and Unicode strings - to non-Unicode if possible, recursively. + Converts subclasses of list and dict to standard lists and dicts, and Unicode strings to + non-Unicode if possible, recursively. Useful for creating human-readable output of structures. """ @@ -216,8 +217,7 @@ def as_agnostic(value): def json_dumps(value, indent=2): """ - JSON dumps that supports Unicode and the :code:`as_raw` property of objects - if available. + JSON dumps that supports Unicode and the ``as_raw`` property of objects if available. """ return json.dumps(value, indent=indent, ensure_ascii=False, cls=JsonAsRawEncoder) @@ -225,8 +225,7 @@ def json_dumps(value, indent=2): def yaml_dumps(value, indent=2): """ - YAML dumps that supports Unicode and the :code:`as_raw` property of objects - if available. + YAML dumps that supports Unicode and the ``as_raw`` property of objects if available. """ return yaml.dump(value, indent=indent, allow_unicode=True, Dumper=YamlAsRawDumper) http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/http.py ---------------------------------------------------------------------- diff --git a/aria/utils/http.py b/aria/utils/http.py index 7bdfd79..c8357e9 100644 --- a/aria/utils/http.py +++ b/aria/utils/http.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +HTTP utilities. +""" + import os import tempfile @@ -20,17 +24,17 @@ import requests def download_file(url, destination=None, logger=None, progress_handler=None): - """Download file. - - May raise IOError as well as requests.exceptions.RequestException - :param url: Location of the file to download - :type url: str - :param destination: - Location where the file should be saved (autogenerated by default) - :type destination: str | None - :returns: Location where the file was saved - :rtype: str + """ + Download file. + :param url: URL from which to download + :type url: basestring + :param destination: path where the file should be saved or ``None`` to auto-generate + :type destination: basestring + :returns: path where the file was saved + :rtype: basestring + :raises exceptions.IOError: + :raises requests.exceptions.RequestException: """ chunk_size = 1024 http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/imports.py ---------------------------------------------------------------------- diff --git a/aria/utils/imports.py b/aria/utils/imports.py index 35aa0fc..14ad09e 100644 --- a/aria/utils/imports.py +++ b/aria/utils/imports.py @@ -14,7 +14,7 @@ # limitations under the License. """ -Utility methods for dynamically loading python code +Utilities for dynamically loading Python code. """ import pkgutil @@ -50,8 +50,8 @@ def import_fullname(name, paths=None): def import_modules(name): """ - Imports a module and all its sub-modules, recursively. - Relies on modules defining a 'MODULES' attribute listing their sub-module names. + Imports a module and all its sub-modules, recursively. Relies on modules defining a ``MODULES`` + attribute listing their sub-module names. """ module = __import__(name, fromlist=['MODULES'], level=0) @@ -63,9 +63,9 @@ def import_modules(name): # TODO merge with import_fullname def load_attribute(attribute_path): """ - Dynamically load an attribute based on the path to it. - e.g. some_package.some_module.some_attribute, will load the some_attribute from the - some_package.some_module module + Dynamically load an attribute based on the path to it. E.g. + ``some_package.some_module.some_attribute``, will load ``some_attribute`` from the + ``some_package.some_module`` module. """ module_name, attribute_name = attribute_path.rsplit('.', 1) try: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/openclose.py ---------------------------------------------------------------------- diff --git a/aria/utils/openclose.py b/aria/utils/openclose.py index 19740eb..722885c 100644 --- a/aria/utils/openclose.py +++ b/aria/utils/openclose.py @@ -13,9 +13,13 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Utilities for working with open/close patterns. +""" + class OpenClose(object): """ - Wraps an object that has open() and close() methods to support the "with" keyword. + Wraps an object that has ``open()`` and ``close()`` methods to support the ``with`` keyword. """ def __init__(self, wrapped): http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/plugin.py ---------------------------------------------------------------------- diff --git a/aria/utils/plugin.py b/aria/utils/plugin.py index b7f94a1..4fb6a8e 100644 --- a/aria/utils/plugin.py +++ b/aria/utils/plugin.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Plugin utilities. +""" + import wagon http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/process.py ---------------------------------------------------------------------- diff --git a/aria/utils/process.py b/aria/utils/process.py index 9aeae67..ec4a72d 100644 --- a/aria/utils/process.py +++ b/aria/utils/process.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Process utilities. +""" + import os http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/specification.py ---------------------------------------------------------------------- diff --git a/aria/utils/specification.py b/aria/utils/specification.py index e74c103..8c51134 100644 --- a/aria/utils/specification.py +++ b/aria/utils/specification.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Utilities for cross-referencing code with specification documents. +""" + from .collections import OrderedDict http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/threading.py ---------------------------------------------------------------------- diff --git a/aria/utils/threading.py b/aria/utils/threading.py index bfd30f5..b9d627a 100644 --- a/aria/utils/threading.py +++ b/aria/utils/threading.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Threading utilities. +""" + from __future__ import absolute_import # so we can import standard 'threading' import sys @@ -26,6 +30,7 @@ from .exceptions import print_exception class ExecutorException(Exception): pass + class DaemonThread(Thread): def __init__(self, *args, **kwargs): super(DaemonThread, self).__init__(*args, **kwargs) @@ -33,16 +38,15 @@ class DaemonThread(Thread): def run(self): """ - We're overriding `Thread.run` in order to avoid annoying (but harmless) error - messages during shutdown. The problem is that CPython nullifies the - global state _before_ shutting down daemon threads, so that exceptions - might happen, and then `Thread.__bootstrap_inner` prints them out. + We're overriding ``Thread.run`` in order to avoid annoying (but harmless) error messages + during shutdown. The problem is that CPython nullifies the global state _before_ shutting + down daemon threads, so that exceptions might happen, and then ``Thread.__bootstrap_inner`` + prints them out. Our solution is to swallow these exceptions here. - The side effect is that uncaught exceptions in our own thread code will _not_ - be printed out as usual, so it's our responsibility to catch them in our - code. + The side effect is that uncaught exceptions in our own thread code will _not_ be printed out + as usual, so it's our responsibility to catch them in our code. """ try: @@ -54,6 +58,7 @@ class DaemonThread(Thread): # Exceptions might occur in daemon threads during interpreter shutdown pass + # https://gist.github.com/tliron/81dd915166b0bfc64be08b4f8e22c835 class FixedThreadPoolExecutor(object): """ @@ -77,8 +82,8 @@ class FixedThreadPoolExecutor(object): executor.raise_first() print executor.returns - You can also use it with the Python "with" keyword, in which case you don't need to call "close" - explicitly:: + You can also use it with the Python ``with`` keyword, in which case you don't need to call + ``close`` explicitly:: with FixedThreadPoolExecutor(10) as executor: for value in range(100): @@ -95,11 +100,10 @@ class FixedThreadPoolExecutor(object): timeout=None, print_exceptions=False): """ - :param size: Number of threads in the pool (fixed). - :param timeout: Timeout in seconds for all - blocking operations. (Defaults to none, meaning no timeout) - :param print_exceptions: Set to true in order to - print exceptions from tasks. (Defaults to false) + :param size: number of threads in the pool; if ``None`` will use an optimal number for the + platform + :param timeout: timeout in seconds for all blocking operations (``None`` means no timeout) + :param print_exceptions: set to ``True`` in order to print exceptions from tasks """ if not size: try: @@ -131,7 +135,7 @@ class FixedThreadPoolExecutor(object): The task will be called ASAP on the next available worker thread in the pool. - Will raise an :class:`ExecutorException` exception if cannot be submitted. + :raises ExecutorException: if cannot be submitted """ try: @@ -145,7 +149,7 @@ class FixedThreadPoolExecutor(object): You cannot submit tasks anymore after calling this. - This is called automatically upon exit if you are using the "with" keyword. + This is called automatically upon exit if you are using the ``with`` keyword. """ self.drain() @@ -194,9 +198,9 @@ class FixedThreadPoolExecutor(object): """ If exceptions were thrown by any task, then the first one will be raised. - This is rather arbitrary: proper handling would involve iterating all the - exceptions. However, if you want to use the "raise" mechanism, you are - limited to raising only one of them. + This is rather arbitrary: proper handling would involve iterating all the exceptions. + However, if you want to use the "raise" mechanism, you are limited to raising only one of + them. """ exceptions = self.exceptions @@ -238,13 +242,13 @@ class FixedThreadPoolExecutor(object): self.close() return False + class LockedList(list): """ - A list that supports the "with" keyword with a built-in lock. + A list that supports the ``with`` keyword with a built-in lock. - Though Python lists are thread-safe in that they will not raise exceptions - during concurrent access, they do not guarantee atomicity. This class will - let you gain atomicity when needed. + Though Python lists are thread-safe in that they will not raise exceptions during concurrent + access, they do not guarantee atomicity. This class will let you gain atomicity when needed. """ def __init__(self, *args, **kwargs): @@ -260,7 +264,7 @@ class LockedList(list): class ExceptionThread(Thread): """ - A thread from which top level exceptions can be retrieved or reraised + A thread from which top level exceptions can be retrieved or re-raised. """ def __init__(self, *args, **kwargs): Thread.__init__(self, *args, **kwargs) http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/2ed2e1c2/aria/utils/type.py ---------------------------------------------------------------------- diff --git a/aria/utils/type.py b/aria/utils/type.py index f08159a..fe88a62 100644 --- a/aria/utils/type.py +++ b/aria/utils/type.py @@ -13,6 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +Type utilities. +""" + import datetime from .specification import implements_specification @@ -79,7 +83,7 @@ def full_type_name(value): @implements_specification('3.2.1-1', 'tosca-simple-1.0') def canonical_type_name(value): """ - Returns the canonical TOSCA type name of a primitive value, or None if unknown. + Returns the canonical TOSCA type name of a primitive value, or ``None`` if unknown. For a list of TOSCA type names, see the `TOSCA Simple Profile v1.0 cos01 specification