atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barbara Eckman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ATLAS-2915) AWS Kinesis Stream Typedef for Atlas
Date Mon, 08 Oct 2018 16:09:00 GMT

     [ https://issues.apache.org/jira/browse/ATLAS-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Barbara Eckman updated ATLAS-2915:
----------------------------------
    Description: 
Currently the base types in Atlas do not include AWS Kinesis Stream objects. It would be
nice to add a typedef for a kinesis stream.  Attributes would include:
 * streamType string, eg ""Single Region Stream".
 * awsRegion string: the AWS region in which the kinesis stream endpoint is deployed
 * shardCount int:  number of shards (uniquely identified sequence of data records) in the
stream
 * streamEnvironment enum.  Valid values are "unknown", "production", "staging", "QA" and
"development"
 * containsPII boolean: does this stream's data contain Personally Identifiable Information?
 * aggregationFormat enum. Indicates if/how the records are aggregated within a single kinesis record.
Valid values are "none" or "kpl".
 * contentType enum: serialization format used by the producer of the stream.  Valid values
are "unknown", "avro", "bson", "csv", "json", "key-value", "kryo", "protobuf", "raw" [ie
no consistent schema], "sdp" [confluent-style avro with envelope that specifies schema id
surrounding the payload], "thrift", "tlv", "xml", "other".
 * schemaURL string: A URL to the data schema used by the producer, to facilitate consumption.
 * avroSchemas: array of avro schema objects (see ATLAS-2694) associated with the kinesis
stream.

 

  was:
Currently the base types in Atlas do not include AWS Kinesis Stream objects. It would be
nice to add a typedef for a kinesis stream.  Attributes would include:
 * streamType string, eg ""Single Region Stream".
 * awsRegion string: the AWS region in which the kinesis stream endpoint is deployed
 * shardCount int:  number of shards (uniquely identified sequence of data records) in the
stream
 * streamEnvironment enum.  Valid values are "unknown", "production", "staging", "QA" and
"development"
 * containsPII boolean: does this stream's data contain Personally Identifiable Information?
 * aggregationFormat enum. Indicates if/how the records are aggregated within a single kinesis record.
Valid values are "none" or "kpl".
 * contentType enum: serialization format used by the producer of the stream.  Valid values
are "unknown", "avro", "bson", "csv", "json", "key-value", "kryo", "protobuf", "raw" [ie
no consistent schema], "sdp" [confluent-style avro with envelope that specifies schema id
surrounding the payload], "thrift", "tlv", "xml", "other".
 * schemaURL string: A URL to the data schema used by the producer, to facilitate consumption.
 * avroSchemas: array of avro schema objects (see ATLAS-XXXX) associated with the kinesis
stream.

 


> AWS Kinesis Stream Typedef for Atlas
> ------------------------------------
>
>                 Key: ATLAS-2915
>                 URL: https://issues.apache.org/jira/browse/ATLAS-2915
>             Project: Atlas
>          Issue Type: New Feature
>            Reporter: Barbara Eckman
>            Priority: Major
>
> Currently the base types in Atlas do not include AWS Kinesis Stream objects. It would
be nice to add a typedef for a kinesis stream.  Attributes would include:
>  * streamType string, eg ""Single Region Stream".
>  * awsRegion string: the AWS region in which the kinesis stream endpoint is deployed
>  * shardCount int:  number of shards (uniquely identified sequence of data records)
in the stream
>  * streamEnvironment enum.  Valid values are "unknown", "production", "staging", "QA"
and "development"
>  * containsPII boolean: does this stream's data contain Personally Identifiable Information?
>  * aggregationFormat enum. Indicates if/how the records are aggregated within a single
kinesis record. Valid values are "none" or "kpl".
>  * contentType enum: serialization format used by the producer of the stream.  Valid
values are "unknown", "avro", "bson", "csv", "json", "key-value", "kryo", "protobuf", "raw"
[ie no consistent schema], "sdp" [confluent-style avro with envelope that specifies schema
id surrounding the payload], "thrift", "tlv", "xml", "other".
>  * schemaURL string: A URL to the data schema used by the producer, to facilitate consumption.
>  * avroSchemas: array of avro schema objects (see ATLAS-2694) associated with the kinesis
stream.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message