gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kam Kasravi (JIRA)" <>
Subject [jira] [Commented] (GEARPUMP-85) Shared Data Types
Date Thu, 21 Apr 2016 17:09:25 GMT


Kam Kasravi commented on GEARPUMP-85:

Imported from [#1457|]

> Shared Data Types
> -----------------
>                 Key: GEARPUMP-85
>                 URL:
>             Project: Apache Gearpump
>          Issue Type: Bug
>          Components: restapi
>    Affects Versions: 0.8.0
>            Reporter: Kam Kasravi
>             Fix For: 0.8.1
> h1. GearPump Shared Data Types
> h2. Definition
> {quote}
> A shared data type is a type that crosses the REST layer either from the client (eg POST)
or to the client (eg GET).  
> {quote}
> h2. Requirements
> # Shared data types should be easily demarked. We should have a way of easily identifying
these data types which is automated. That is - a programmatic way of finding these data types
reliably that requires no separate list or manual maintenance which could become outdated.
> # The REST API should have documentation auto-generated. This includes the endpoints
and the shared data types. It should be part of the build process.
> # All shared data types should be JSON friendly. That is no data type should require
special serde (serialization/deserialization) methods on client or server to handle unserializable
> # There should be compilation checks that prevent type related errors. That is changing
a shared data type on either the client or server will result in a compilation error on the
flip side.
> # Shared data types should not need to be repackaged. Current types should be able to
remain as is, new shared data types should be able to be defined where appropriate.
> # Annotations or other constructs should be minimal. Given the constraint that shared
data types can be defined anywhere, there needs to be a way to identify them. Naming conventions
are not a possibility due to no naming convention has been followed to date. If annotations
are used they should be minimal.
> # Shared data types should provide a [json schema (v4)](
or equivalent that allows tooling to transpile these shared data types to potentially other
languages. 3rd party clients may require a schema to develop any reasonable client-side functionality
within reasonable time frames.
> h2. Use Cases
> GearPump provides a REST API that is used to expose a wide variety of functionality both
for the user of GearPump and the administrator. The REST API leverages JSON in many HTTP GET,
POST, PUT, DELETE operations. These JSON structures are used in javascript as javascript objects,
within the REST server and within the master, worker and executor actors. They permeate all
environments. Given their scope and prevalence we should have a set of guidelines on how these
data types should be created, changed and transported. This will minimize a variety of problems.

> h2. Current usage 
> Currently shared data types as defined above are found across a large number of packages
on the server and inline with other classes, objects and other scala definitions. There is
no devoted package or equivalent construct to isolate these data types. They are also not
annotated or otherwise identified within the code base. That is - there is no way to reliably
gather these types other than a tedious review of the services classes and their route paths.
Additionally, within the client layer there is no typing of these shared data types within
javascript. For GET operations these data types are reified into untyped javascript object
literals, where field naming is preserved but type naming is not. In other words a scala data
type of `case class MasterData(masterDescription: MasterSummary)` is not identified by that
name in the client code. There is also no tooling in place to auto-generate documentation
for these shared data types or the REST API in general. Any 3rd party wishing to build their
own client would need to scour the code and attempt to cobble together the REST endpoints
and their related data. This would be untenable for most 3rd parties and represents a significant
obstacle in promoting GearPump's many other strengths in a very competitive landscape.
> h3. Positives
> There are some things that are done on the server with these shared data types that are
> # upickle is used to read and write all shared data types on the server. (Unfortunately
upickle is not used on the client).
> # There is minimal specialized type conversions. Upickle provides a way to take a type
that cannot normally be serialized and allows one to provide a customized JSON mapping. For
example Actor is not a serializable data type, if one were to serialize an Actor using upickle
they would need to provide a customized serializer/deserializer for it. There are few places
(if any) where we do this.
> # All data types are case classes.
> h3. Suggested Design approaches
> Design solutions should attempt to leverage best-in-class tooling - for example for REST
related documentation. Any design solution should not tax the current work flow with a heavy
weight process related to development on either the client or server side.

This message was sent by Atlassian JIRA

View raw message