Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id ABAF7200CB5 for ; Wed, 28 Jun 2017 07:57:57 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id AA05F160BE9; Wed, 28 Jun 2017 05:57:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5F637160BDC for ; Wed, 28 Jun 2017 07:57:55 +0200 (CEST) Received: (qmail 59592 invoked by uid 500); 28 Jun 2017 05:57:54 -0000 Mailing-List: contact commits-help@atlas.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.apache.org Delivered-To: mailing list commits@atlas.apache.org Received: (qmail 59583 invoked by uid 99); 28 Jun 2017 05:57:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jun 2017 05:57:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id C78DBC061C for ; Wed, 28 Jun 2017 05:57:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.212 X-Spam-Level: X-Spam-Status: No, score=-4.212 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, T_FILL_THIS_FORM_SHORT=0.01] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id uMmk-fQ3IwIG for ; Wed, 28 Jun 2017 05:57:38 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id DD24860E17 for ; Wed, 28 Jun 2017 05:57:18 +0000 (UTC) Received: (qmail 56937 invoked by uid 99); 28 Jun 2017 05:57:16 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jun 2017 05:57:16 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 9399EF3249; Wed, 28 Jun 2017 05:57:15 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: madhan@apache.org To: commits@atlas.incubator.apache.org Date: Wed, 28 Jun 2017 05:57:36 -0000 Message-Id: <9031c2b797804e24ad0447ee60771b5b@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [23/25] incubator-atlas git commit: ATLAS-1898: initial commit of ODF archived-at: Wed, 28 Jun 2017 05:57:57 -0000 http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/README.md ---------------------------------------------------------------------- diff --git a/odf/README.md b/odf/README.md new file mode 100755 index 0000000..fbacd07 --- /dev/null +++ b/odf/README.md @@ -0,0 +1,20 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +Open Discovery Framework +========================== + +The Open Discovery Framework (ODF) is an open metadata-based framework that strives to be a common home for different analytics technologies that discover characteristics of data sets and relationships between them (think "AppStore for discovery algorithms"). Using ODF, applications can leverage new discovery algorithms and their results with minimal integration effort. + +See [here](odf-doc/src/site/markdown/build.md) how to build and deploy ODF. http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/jettyconfig/jetty-https.xml ---------------------------------------------------------------------- diff --git a/odf/jettyconfig/jetty-https.xml b/odf/jettyconfig/jetty-https.xml new file mode 100755 index 0000000..283e511 --- /dev/null +++ b/odf/jettyconfig/jetty-https.xml @@ -0,0 +1,63 @@ + + + + + + + + + + + + + + + http/1.1 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ODF Realm + /realm.properties + + + + http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/jettyconfig/jetty-ssl.xml ---------------------------------------------------------------------- diff --git a/odf/jettyconfig/jetty-ssl.xml b/odf/jettyconfig/jetty-ssl.xml new file mode 100755 index 0000000..fb5b5e3 --- /dev/null +++ b/odf/jettyconfig/jetty-ssl.xml @@ -0,0 +1,45 @@ + + + + + /keystore.jks + OBF:20zh1zsv1kjo1lca1lf81kmy1zsv20zl + OBF:20zh1zsv1kjo1lca1lf81kmy1zsv20zl + /keystore.jks + OBF:20zh1zsv1kjo1lca1lf81kmy1zsv20zl + + + + SSL_RSA_WITH_DES_CBC_SHA + SSL_DHE_RSA_WITH_DES_CBC_SHA + SSL_DHE_DSS_WITH_DES_CBC_SHA + SSL_RSA_EXPORT_WITH_RC4_40_MD5 + SSL_RSA_EXPORT_WITH_DES40_CBC_SHA + SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA + SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA + + + + + + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/jettyconfig/jetty.xml ---------------------------------------------------------------------- diff --git a/odf/jettyconfig/jetty.xml b/odf/jettyconfig/jetty.xml new file mode 100755 index 0000000..c754b48 --- /dev/null +++ b/odf/jettyconfig/jetty.xml @@ -0,0 +1,28 @@ + + + + + + + + https + + 32768 + 8192 + 8192 + true + false + 512 + http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/jettyconfig/realm.properties ---------------------------------------------------------------------- diff --git a/odf/jettyconfig/realm.properties b/odf/jettyconfig/realm.properties new file mode 100755 index 0000000..109d726 --- /dev/null +++ b/odf/jettyconfig/realm.properties @@ -0,0 +1,24 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Credentials for ODF basic authentication +# +# Format: +# : [, ...] +# +# Password is stored in obfuscated format. +# Re-generate password using the org.eclipse.jetty.util.security.Password class in the jetty lib folder. +# Example: +# cd jetty-distribution-/lib +# java -cp jetty-util-.jar org.eclipse.jetty.util.security.Password +sdp: OBF:1ugg1sov1xfd1k8k1wn31k5m1xfp1sov1uha,user http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/.gitignore ---------------------------------------------------------------------- diff --git a/odf/odf-api/.gitignore b/odf/odf-api/.gitignore new file mode 100755 index 0000000..2daccfd --- /dev/null +++ b/odf/odf-api/.gitignore @@ -0,0 +1,18 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +.settings +target +.classpath +.project +.factorypath http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/pom.xml ---------------------------------------------------------------------- diff --git a/odf/odf-api/pom.xml b/odf/odf-api/pom.xml new file mode 100755 index 0000000..5c8258d --- /dev/null +++ b/odf/odf-api/pom.xml @@ -0,0 +1,100 @@ + + + + 4.0.0 + + org.apache.atlas.odf + odf + 1.2.0-SNAPSHOT + + odf-api + + UTF-8 + + + + com.fasterxml.jackson.core + jackson-annotations + ${jackson.version} + compile + + + com.fasterxml.jackson.core + jackson-databind + ${jackson.version} + compile + + + org.apache.wink + wink-json4j + 1.4 + + + org.apache.commons + commons-csv + 1.2 + + + junit + junit + 4.12 + test + + + org.hamcrest + hamcrest-all + 1.3 + + + org.apache.httpcomponents + fluent-hc + 4.5.1 + compile + + + swagger-jaxrs + 1.5.9 + io.swagger + compile + + + + org.apache.spark + spark-core_2.11 + 2.1.0 + provided + + + commons-codec + commons-codec + + + + + org.apache.spark + spark-sql_2.11 + 2.1.0 + provided + + + commons-codec + commons-codec + + + + + http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/ODFFactory.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/ODFFactory.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/ODFFactory.java new file mode 100755 index 0000000..20676b4 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/ODFFactory.java @@ -0,0 +1,41 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api; + +import java.text.MessageFormat; + +public class ODFFactory { + + private final static String ODF_DEFAULT_IMPLEMENTATION = "org.apache.atlas.odf.core.OpenDiscoveryFrameworkImpl"; + + public OpenDiscoveryFramework create() { + Object o = null; + Class clazz; + try { + clazz = this.getClass().getClassLoader().loadClass(ODF_DEFAULT_IMPLEMENTATION); + } catch (ClassNotFoundException e) { + throw new RuntimeException(MessageFormat.format("Class {0} was not found. Make sure that the odf-core jar and all its dependencies are available on the classpath.", ODF_DEFAULT_IMPLEMENTATION)); + } + try { + o = clazz.newInstance(); + } catch (InstantiationException | IllegalAccessException e) { + throw new RuntimeException(MessageFormat.format("Class {0} was found on the classpath but could not be accessed.", ODF_DEFAULT_IMPLEMENTATION)); + } + if (o instanceof OpenDiscoveryFramework) { + return (OpenDiscoveryFramework) o; + } else { + throw new RuntimeException(MessageFormat.format("The class {0} on the classpath is not of type OpenDiscoveryFramework.", ODF_DEFAULT_IMPLEMENTATION)); + } + } +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/OpenDiscoveryFramework.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/OpenDiscoveryFramework.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/OpenDiscoveryFramework.java new file mode 100755 index 0000000..70ab91b --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/OpenDiscoveryFramework.java @@ -0,0 +1,79 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api; + +import org.apache.atlas.odf.api.analysis.AnalysisManager; +import org.apache.atlas.odf.api.annotation.AnnotationStore; +import org.apache.atlas.odf.api.discoveryservice.DiscoveryServiceManager; +import org.apache.atlas.odf.api.engine.EngineManager; +import org.apache.atlas.odf.api.metadata.MetadataStore; +import org.apache.atlas.odf.api.metadata.importer.JDBCMetadataImporter; +import org.apache.atlas.odf.api.settings.SettingsManager; + +/** +* +* External Java API for managing and controlling ODF +* +*/ +public interface OpenDiscoveryFramework { + + /** + * Returns API for managing ODF analysis requests + * + * @return ODF analysis manager API + */ + public AnalysisManager getAnalysisManager(); + + /** + * Returns API for managing ODF discovery services + * + * @return ODF discovery services manager API + */ + public DiscoveryServiceManager getDiscoveryServiceManager(); + + /** + * Returns API for controlling the ODF engine + * + * @return ODF engine manager API + */ + public EngineManager getEngineManager(); + + /** + * Returns API for managing ODF settings + * + * @return ODF settings manager API + */ + public SettingsManager getSettingsManager(); + + /** + * Returns ODF annotation store API + * + * @return ODF annotation store API + */ + public AnnotationStore getAnnotationStore(); + + /** + * Returns ODF metadata store API + * + * @return ODF metadata store API + */ + public MetadataStore getMetadataStore(); + + /** + * Returns JDBC importer utility for populating the metadata store with sample data + * + * @return ODF JDBC importer utility + */ + public JDBCMetadataImporter getJDBCMetadataImporter(); +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisCancelResult.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisCancelResult.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisCancelResult.java new file mode 100755 index 0000000..cd294c5 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisCancelResult.java @@ -0,0 +1,34 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +public class AnalysisCancelResult { + + public enum State { + NOT_FOUND, + INVALID_STATE, + SUCCESS + } + + private State state; + + public State getState() { + return state; + } + + public void setState(State state) { + this.state = state; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisManager.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisManager.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisManager.java new file mode 100755 index 0000000..6ff6098 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisManager.java @@ -0,0 +1,62 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +/** + * + * External interface for creating and managing analysis requests + * + */ +public interface AnalysisManager { + + /** + * Issues a new ODF analysis request + * + * @param request Analysis request + * @return Response containing the request id and status information + */ + public AnalysisResponse runAnalysis(AnalysisRequest request); + + /** + * Retrieve status of an ODF analysis request + * + * @param requestId Unique id of the analysis request + * @return Status of the analysis request + */ + public AnalysisRequestStatus getAnalysisRequestStatus(String requestId); + + /** + * Retrieve statistics about all previous ODF analysis requests + * + * @return Request summary + */ + public AnalysisRequestSummary getAnalysisStats(); + + /** + * Retrieve status details of recent ODF analysis requests + * + * @param offset Starting offset (use 0 to start with the latest request) + * @param limit Maximum number of analysis requests to be returned (use -1 to retrieve all requests) + * @return Status details for each discovery request + */ + public AnalysisRequestTrackers getAnalysisRequests(int offset, int limit); + + /** + * Request a specific ODF discovery request to be canceled + * + * @param requestId Unique id of the analysis request + * @return Status of the cancellation attempt + */ + public AnalysisCancelResult cancelAnalysisRequest(String requestId); +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequest.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequest.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequest.java new file mode 100755 index 0000000..3aa5937 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequest.java @@ -0,0 +1,108 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.atlas.odf.api.metadata.MetaDataObjectReference; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; + +// JSON +@ApiModel(description="Request for starting a discovery service.") +public class AnalysisRequest { + + // only used when returned by the ODF + @ApiModelProperty(value="Unique request id (generated)", readOnly=true, required=false) + private String id; + + @ApiModelProperty(value="Data set to be analyzed (currently limited to a single data set)", required=true) + private List dataSets = new ArrayList<>(); + + @ApiModelProperty(value="Sequence of ids (or single id) of the discovery services to be issued", required=false) + private List discoveryServiceSequence = new ArrayList(); + + @ApiModelProperty(value="List annotation types to be created on the dataset(s)", required=false) + private List annotationTypes = new ArrayList(); + + @ApiModelProperty(value="Optional additional properties map to be passed to the discovery service(s)", required=false) + private Map additionalProperties = new HashMap(); + + @ApiModelProperty(value="Indicates that multiple data sets should be processed sequentially rather than in parallel", required=false) + private boolean processDataSetsSequentially = true; + + // if false the request will fail if some discovery service that cannot process a data set + @ApiModelProperty(value="Indicates that access to the data set should not be checked before starting the discovery service", required=false) + private boolean ignoreDataSetCheck = false; + + public List getDataSets() { + return dataSets; + } + + public void setDataSets(List dataSets) { + this.dataSets = dataSets; + } + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public List getDiscoveryServiceSequence() { + return discoveryServiceSequence; + } + + public void setDiscoveryServiceSequence(List discoveryServiceSequence) { + this.discoveryServiceSequence = discoveryServiceSequence; + } + + public List getAnnotationTypes() { + return annotationTypes; + } + + public void setAnnotationTypes(List annotationTypes) { + this.annotationTypes = annotationTypes; + } + + public Map getAdditionalProperties() { + return additionalProperties; + } + + public void setAdditionalProperties(Map additionalProperties) { + this.additionalProperties = additionalProperties; + } + + public boolean isProcessDataSetsSequentially() { + return processDataSetsSequentially; + } + + public void setProcessDataSetsSequentially(boolean processDataSetsSequentially) { + this.processDataSetsSequentially = processDataSetsSequentially; + } + + public boolean isIgnoreDataSetCheck() { + return ignoreDataSetCheck; + } + + public void setIgnoreDataSetCheck(boolean ignoreDataSetCheck) { + this.ignoreDataSetCheck = ignoreDataSetCheck; + } +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestStatus.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestStatus.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestStatus.java new file mode 100755 index 0000000..b6a120e --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestStatus.java @@ -0,0 +1,124 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +import java.util.List; + +import org.apache.atlas.odf.api.discoveryservice.DiscoveryServiceRequest; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; + +// JSON +@ApiModel(description="Status of a specific analysis request.") +public class AnalysisRequestStatus { + + public static enum State { + ACTIVE, // some discovery service is processing the request + QUEUED, // in the queue for some discovery service + ERROR, // something went wrong + FINISHED, // processing finished successfully + NOT_FOUND, // request ID was not found + CANCELLED // request was cancelled by the user + } + + @ApiModelProperty(value="Analysis request that was submitted", readOnly=true, required=true) + private AnalysisRequest request; + + @ApiModelProperty(value="Status of the request", readOnly=true, required=true) + private State state; + + @ApiModelProperty(value="Detailed status description", readOnly=true, required=false) + private String details; + + @ApiModelProperty(value="Indicates whether an equivalent request was found", readOnly=true, required=true) + private boolean foundExistingRequest = false; + + @ApiModelProperty(value="List of individual discovery service requests that make up the analysis request", readOnly=true, required=true) + private List serviceRequests; + + @ApiModelProperty(value="Total time the request was queued in milliseconds", readOnly=true, required=true) + private long totalTimeOnQueues; + + @ApiModelProperty(value="Total time needed for processing the analysis request in milliseconds", readOnly=true, required=true) + private long totalTimeProcessing; + + @ApiModelProperty(value="Total time needed for storing the annotations in the metadata repository in milliseconds", readOnly=true, required=true) + private long totalTimeStoringAnnotations; + + public AnalysisRequest getRequest() { + return request; + } + + public void setRequest(AnalysisRequest request) { + this.request = request; + } + + public State getState() { + return state; + } + + public void setState(State state) { + this.state = state; + } + + public String getDetails() { + return details; + } + + public void setDetails(String details) { + this.details = details; + } + + public boolean isFoundExistingRequest() { + return foundExistingRequest; + } + + public void setFoundExistingRequest(boolean foundExistingRequest) { + this.foundExistingRequest = foundExistingRequest; + } + + public List getServiceRequests() { + return serviceRequests; + } + + public void setServiceRequests(List requests) { + this.serviceRequests = requests; + } + + public long getTotalTimeOnQueues() { + return totalTimeOnQueues; + } + + public void setTotalTimeOnQueues(long totalTimeOnQueues) { + this.totalTimeOnQueues = totalTimeOnQueues; + } + + public long getTotalTimeProcessing() { + return totalTimeProcessing; + } + + public void setTotalTimeProcessing(long totalTimeProcessing) { + this.totalTimeProcessing = totalTimeProcessing; + } + + public long getTotalTimeStoringAnnotations() { + return totalTimeStoringAnnotations; + } + + public void setTotalTimeStoringAnnotations(long totalTimeStoringAnnotations) { + this.totalTimeStoringAnnotations = totalTimeStoringAnnotations; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestSummary.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestSummary.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestSummary.java new file mode 100755 index 0000000..a7982ef --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestSummary.java @@ -0,0 +1,52 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; + +@ApiModel(description="Status summary of all analysis requests submitted since last start of ODF.") +public class AnalysisRequestSummary { + + @ApiModelProperty(value="Number of successful analysis requests", readOnly=true, required=true) + private int success; + + @ApiModelProperty(value="Number of failing analysis requests", readOnly=true, required=true) + private int failure; + + AnalysisRequestSummary() { + } + + public AnalysisRequestSummary(int success, int failure) { + this.success = success; + this.failure = failure; + } + + public int getSuccess() { + return this.success; + } + + public void setSuccess(int success) { + this.success = success; + } + + public int getFailure() { + return this.failure; + } + + public void setFailure(int failure) { + this.failure = failure; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackerStatus.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackerStatus.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackerStatus.java new file mode 100755 index 0000000..49ca9f7 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackerStatus.java @@ -0,0 +1,25 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +public class AnalysisRequestTrackerStatus { + public static enum STATUS { + INITIALIZED, //tracker was created, nothing else happened so far + IN_DISCOVERY_SERVICE_QUEUE, //tracker is put on queue but not running yet + DISCOVERY_SERVICE_RUNNING, //only for async services, analysis is running + FINISHED, //analysis finished + ERROR, // an error occurred during analysis / processing + CANCELLED //the analysis was cancelled by the user + }; +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackers.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackers.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackers.java new file mode 100755 index 0000000..846ed3d --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisRequestTrackers.java @@ -0,0 +1,36 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +import java.util.List; + +import org.apache.atlas.odf.api.discoveryservice.AnalysisRequestTracker; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; + +@ApiModel(description="Container object tracking the status of multiple analysis requests.") +public class AnalysisRequestTrackers { + + @ApiModelProperty(value="List of container objects tracking the status of analysis requests", required=true) + private List analysisRequestTrackers; + + public List getAnalysisRequestTrackers() { + return this.analysisRequestTrackers; + } + + public void setAnalysisRequestTrackers(List analysisRequestTrackers) { + this.analysisRequestTrackers = analysisRequestTrackers; + } +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisResponse.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisResponse.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisResponse.java new file mode 100755 index 0000000..9f1e45c --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/analysis/AnalysisResponse.java @@ -0,0 +1,66 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.analysis; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; + +// JSON +@ApiModel(description="Response returned by the analysis request.") +public class AnalysisResponse { + @ApiModelProperty(value="Unique request id", readOnly=true, required=true) + private String id; + + @ApiModelProperty(value="Original request that is equivalent to the submitted one which is therefore skipped", readOnly=true, required=true) + private AnalysisRequest originalRequest; + + private boolean isInvalidRequest = false; + + @ApiModelProperty(value="Details about why the request is invalid", readOnly=true, required=false) + private String details; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public AnalysisRequest getOriginalRequest() { + return originalRequest; + } + + public void setOriginalRequest(AnalysisRequest originalRequest) { + this.originalRequest = originalRequest; + } + + @ApiModelProperty(name="isInvalidRequest", value="Indicates whether the submitted request is invalid", readOnly=true, required=true) + public boolean isInvalidRequest() { + return isInvalidRequest; + } + + public void setInvalidRequest(boolean isInvalidRequest) { + this.isInvalidRequest = isInvalidRequest; + } + + public String getDetails() { + return details; + } + + public void setDetails(String details) { + this.details = details; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStore.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStore.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStore.java new file mode 100755 index 0000000..7e08d74 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStore.java @@ -0,0 +1,43 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.annotation; + +import java.util.List; + +import org.apache.atlas.odf.api.metadata.ExternalStore; +import org.apache.atlas.odf.api.metadata.MetaDataObjectReference; +import org.apache.atlas.odf.api.metadata.models.Annotation; + +public interface AnnotationStore extends ExternalStore { + + /** + * @return the reference to the object that was created or updated + */ + MetaDataObjectReference store(Annotation annotation); + + /** + * Get all annotations attached to the meta data object for a specific analysis request. + */ + List getAnnotations(MetaDataObjectReference object, String analysisRequestId); + + /** + * Retrieve an annotation by ID + */ + Annotation retrieveAnnotation(MetaDataObjectReference ref); + + /// internal + void setAnalysisRun(String analysisRun); + + String getAnalysisRun(); +}; http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStoreUtils.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStoreUtils.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStoreUtils.java new file mode 100755 index 0000000..e51faf6 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/AnnotationStoreUtils.java @@ -0,0 +1,129 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.annotation; + +import java.text.MessageFormat; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.logging.Level; +import java.util.logging.Logger; + +import org.apache.atlas.odf.api.metadata.MetaDataObjectReference; +import org.apache.atlas.odf.api.metadata.MetadataStoreException; +import org.apache.atlas.odf.api.metadata.models.Annotation; +import org.apache.atlas.odf.api.metadata.models.ClassificationAnnotation; +import org.apache.atlas.odf.api.metadata.models.MetaDataObject; +import org.apache.atlas.odf.api.metadata.models.ProfilingAnnotation; +import org.apache.atlas.odf.api.metadata.models.RelationshipAnnotation; + +public class AnnotationStoreUtils { + + /** + * Return the most recent annotations for the passed object but at most one per annotation type. + * Note that this might not be suitable for the semantics represented by some annotation types. + */ + public static List getMostRecentAnnotationsByType(AnnotationStore as, MetaDataObjectReference object) { + try { + // Fix issue 99: only return one annotation per type + Map mostRecentAnnotationsByType = new HashMap<>(); + Map typeToMaxTimestamp = new HashMap<>(); + for (Annotation annot : as.getAnnotations(object, null)) { + Long ts = getTimestamp(annot); + String annotType = annot.getAnnotationType(); + Long l = typeToMaxTimestamp.get(annotType); + if (l == null) { + l = ts; + } + if (l <= ts) { + typeToMaxTimestamp.put(annotType, Long.valueOf(ts)); + mostRecentAnnotationsByType.put(annotType, annot); + } + } + return new ArrayList<>(mostRecentAnnotationsByType.values()); + } catch (Exception exc) { + throw new MetadataStoreException(exc); + } + } + + private static long getTimestamp(Annotation annot) { + final long defaultVal = -1; + String runId = annot.getAnalysisRun(); + int ix = runId.lastIndexOf("_"); + if (ix == -1) { + return defaultVal; + } + String millis = runId.substring(ix); + long result = defaultVal; + try { + result = Long.valueOf(millis); + } catch (NumberFormatException e) { + return defaultVal; + } + return result; + } + + /** + * Retrieve the annotations of a meta data object for a given annotation store, and annotation type that have been created by + * a certain request. + * + * @param mdo + * @param store + * @param annotationType + * @param requestId + * @return List of annotations for this mdo, annotation store, and annotation type created by the request with the ID 'requestId' + */ + + public static List retrieveAnnotationsOfRun(MetaDataObject mdo, AnnotationStore store, String annotationType, String requestId) { + Logger logger = Logger.getLogger(AnnotationStoreUtils.class.getName()); + List annotations = new ArrayList<>(); + for (Annotation annot : store.getAnnotations(mdo.getReference(), null)) { + logger.log(Level.FINER, "Found annotation on object {0} with analysis run {1} and annotationType {2}", + new Object[] { mdo.getReference().getId(), annot.getAnalysisRun(), annot.getAnnotationType() }); + if (annot.getAnalysisRun().equals(requestId) && annot.getAnnotationType().equals(annotationType)) { + annotations.add(annot); + } + } + return annotations; + } + + /** + * For a given annotation return the reference to the annotated object. Throw a MetaDataStoreException if this reference is null. + * + * @param annot + * @return Meta data reference of annotation 'annot' + */ + + public static MetaDataObjectReference getAnnotatedObject(Annotation annot) { + MetaDataObjectReference annotRef = null; + if (annot instanceof ProfilingAnnotation) { + annotRef = ((ProfilingAnnotation) annot).getProfiledObject(); + } else if (annot instanceof ClassificationAnnotation) { + annotRef = ((ClassificationAnnotation) annot).getClassifiedObject(); + } else if (annot instanceof RelationshipAnnotation) { + + List refs = ((RelationshipAnnotation) annot).getRelatedObjects(); + if (refs != null && refs.size() > 0) { + annotRef = refs.get(0); + } + } + if (annotRef == null) { + String errorMessage = MessageFormat.format("The annotated object of annotation with ID ''{0}'' is null.", annot.getReference().getId()); + throw new MetadataStoreException(errorMessage); + } + return annotRef; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/Annotations.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/Annotations.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/Annotations.java new file mode 100755 index 0000000..058b472 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/annotation/Annotations.java @@ -0,0 +1,31 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.annotation; + +import java.util.List; + +import org.apache.atlas.odf.api.metadata.models.Annotation; + +public class Annotations { + private List annotations; + + public List getAnnotations() { + return annotations; + } + + public void setAnnotations(List annotations) { + this.annotations = annotations; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataRetrievalException.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataRetrievalException.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataRetrievalException.java new file mode 100755 index 0000000..c67ae97 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataRetrievalException.java @@ -0,0 +1,35 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.connectivity; + +public class DataRetrievalException extends RuntimeException { + + /** + * + */ + private static final long serialVersionUID = 4978058839277657L; + + public DataRetrievalException() { + super(); + } + + public DataRetrievalException(String message) { + super(message); + } + + public DataRetrievalException(Throwable cause) { + super(cause); + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetriever.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetriever.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetriever.java new file mode 100755 index 0000000..8dc2579 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetriever.java @@ -0,0 +1,33 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.connectivity; + +import org.apache.atlas.odf.api.metadata.MetadataStore; +import org.apache.atlas.odf.api.metadata.models.DataSet; +import org.apache.atlas.odf.api.metadata.models.RelationalDataSet; +import org.apache.atlas.odf.api.metadata.models.Table; +import org.apache.atlas.odf.api.discoveryservice.datasets.MaterializedDataSet; + +public interface DataSetRetriever { + + void setMetadataStore(MetadataStore mds); + + boolean canRetrieveDataSet(DataSet oMDataSet); + + MaterializedDataSet retrieveRelationalDataSet(RelationalDataSet relationalDataSet) throws DataRetrievalException; + + void createCsvFile(RelationalDataSet relationalDataSet, String fileName) throws DataRetrievalException; + + JDBCRetrievalResult retrieveTableAsJDBCResultSet(Table oMTable) throws DataRetrievalException; +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetrieverImpl.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetrieverImpl.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetrieverImpl.java new file mode 100755 index 0000000..6846ba5 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/DataSetRetrieverImpl.java @@ -0,0 +1,298 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.connectivity; + +import java.io.File; +import java.io.PrintWriter; +import java.net.URL; +import java.nio.charset.Charset; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.text.MessageFormat; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.logging.Level; +import java.util.logging.Logger; + +import org.apache.atlas.odf.api.metadata.MetadataStore; +import org.apache.commons.csv.CSVFormat; +import org.apache.commons.csv.CSVParser; +import org.apache.commons.csv.CSVRecord; + +import org.apache.atlas.odf.api.metadata.models.JDBCConnection; +import org.apache.atlas.odf.api.metadata.models.JDBCConnectionInfo; +import org.apache.atlas.odf.api.metadata.models.Column; +import org.apache.atlas.odf.api.metadata.models.DataFile; +import org.apache.atlas.odf.api.metadata.models.DataSet; +import org.apache.atlas.odf.api.metadata.models.RelationalDataSet; +import org.apache.atlas.odf.api.metadata.models.Table; +import org.apache.atlas.odf.api.discoveryservice.datasets.MaterializedDataSet; + +/** + * This class is a helper to retrieve actual data from a data source by passing an object that represents a reference to the dataset. + * + */ +public class DataSetRetrieverImpl implements DataSetRetriever { + + Logger logger = Logger.getLogger(DataSetRetrieverImpl.class.getName()); + MetadataStore metaDataStore; + + public DataSetRetrieverImpl() { + } + + public DataSetRetrieverImpl(MetadataStore metaDataStore) { + this.metaDataStore = metaDataStore; + } + + @Override + public void setMetadataStore(MetadataStore mds) { + this.metaDataStore = mds; + } + + @Override + public boolean canRetrieveDataSet(DataSet dataSet) { + if (dataSet instanceof DataFile) { + DataFile dataFile = (DataFile) dataSet; + return getValidURL(dataFile) != null; + } else if (dataSet instanceof Table) { + Connection connection = getJDBCConnection((JDBCConnectionInfo) metaDataStore.getConnectionInfo(dataSet)); + if (connection != null) { + try { + connection.close(); + return true; + } catch (SQLException e) { + // do nothing + } + } + } + return false; + } + + @Override + public MaterializedDataSet retrieveRelationalDataSet(RelationalDataSet relationalDataSet) { + if (relationalDataSet instanceof DataFile) { + return retrieveDataFile((DataFile) relationalDataSet); + } else if (relationalDataSet instanceof Table) { + return retrieveTableWithJDBC((Table) relationalDataSet); + } + return null; + } + + @Override + public void createCsvFile(RelationalDataSet relationalDataSet, String fileName) { + try { + logger.log(Level.INFO, "Creating CSV input data file ", fileName); + MaterializedDataSet mds = retrieveRelationalDataSet(relationalDataSet); + PrintWriter printWriter = new PrintWriter(new File(fileName), "UTF-8") ; + int columnCount = mds.getColumns().size(); + String headers = "\"" + mds.getColumns().get(0).getName() + "\"" ; + for (int i = 1; i < columnCount; i++) { + headers += ",\"" + mds.getColumns().get(i).getName() + "\"" ; + } + printWriter.println(headers); + for (int i = 0; i < mds.getData().size(); i++) { + String row = "\"" + mds.getData().get(i).get(0).toString() + "\""; + for (int j = 1 ; j < columnCount; j++ ) { + row += ",\"" + mds.getData().get(i).get(j).toString() + "\""; + } + printWriter.println(row); + } + printWriter.close(); + } catch(Exception exc) { + throw new DataRetrievalException(exc); + } + } + + private URL getValidURL(DataFile dataFile) { + try { + Charset.forName(dataFile.getEncoding()); + } catch (Exception exc) { + logger.log(Level.WARNING, MessageFormat.format("Encoding ''{0}'' of data file ''{1}''is not valid''", new Object[] { dataFile.getEncoding(), dataFile.getUrlString() }), exc); + return null; + } + String urlString = dataFile.getUrlString(); + try { + URL url = new URL(urlString); + url.openConnection().connect(); + return url; + } catch (Exception exc) { + String msg = MessageFormat.format("Could not connect to data file URL ''{0}''. Error: {1}", new Object[] { urlString, exc.getMessage() }); + logger.log(Level.WARNING, msg, exc); + return null; + } + } + + private MaterializedDataSet retrieveDataFile(DataFile dataFile) throws DataRetrievalException { + URL url = this.getValidURL(dataFile); + if (url == null) { + return null; + } + List columns = metaDataStore.getColumns(dataFile); + List> data = new ArrayList<>(); + + try { + CSVParser csvParser = CSVParser.parse(url, Charset.forName(dataFile.getEncoding()), CSVFormat.DEFAULT.withHeader()); + List records = csvParser.getRecords(); + Map headerMap = csvParser.getHeaderMap(); + csvParser.close(); + + for (CSVRecord record : records) { + List targetRecord = new ArrayList<>(); + for (int i = 0; i < columns.size(); i++) { + Column col = columns.get(i); + String value = record.get(headerMap.get(col.getName())); + Object convertedValue = value; + if (col.getDataType().equals("int")) { + convertedValue = Integer.parseInt(value); + } else if (col.getDataType().equals("double")) { + convertedValue = Double.parseDouble(value); + } + // TODO add more conversions + targetRecord.add(convertedValue); + } + data.add(targetRecord); + } + + } catch (Exception exc) { + throw new DataRetrievalException(exc); + } + + MaterializedDataSet materializedDS = new MaterializedDataSet(); + materializedDS.setTable(dataFile); + materializedDS.setColumns(columns); + materializedDS.setData(data); + return materializedDS; + } + + public static String quoteForJDBC(String s) { + // TODO implement to prevent SQL injection + return s; + } + + Connection getJDBCConnection(JDBCConnectionInfo connectionInfo) { + if ((connectionInfo.getConnections() == null) || connectionInfo.getConnections().isEmpty()) { + return null; + } + JDBCConnection connectionObject = null; + connectionObject = (JDBCConnection) connectionInfo.getConnections().get(0); // Use first connection + try { + return DriverManager.getConnection(connectionObject.getJdbcConnectionString(), connectionObject.getUser(), connectionObject.getPassword()); + } catch (SQLException exc) { + logger.log(Level.WARNING, MessageFormat.format("JDBC connection to ''{0}'' for table ''{1}'' could not be created", new Object[] { connectionObject.getJdbcConnectionString(), + connectionInfo.getSchemaName() + "." + connectionInfo.getTableName() }), exc); + return null; + } + } + + private MaterializedDataSet retrieveTableWithJDBC(Table table) { + + JDBCRetrievalResult jdbcRetrievalResult = this.retrieveTableAsJDBCResultSet(table); + if (jdbcRetrievalResult == null) { + logger.log(Level.FINE, "JDBC retrieval result for table ''{0}'' is null", table.getReference().getUrl()); + return null; + } + + Map columnMap = new HashMap(); + for (Column column : metaDataStore.getColumns(table)) { + columnMap.put(column.getName(), column); + } + logger.log(Level.INFO, "Table columns: {0}", columnMap.keySet()); + + ResultSet rs = null; + try { + logger.log(Level.FINE, "Executing prepared statement " + jdbcRetrievalResult.getPreparedStatement()); + rs = jdbcRetrievalResult.getPreparedStatement().executeQuery(); + ResultSetMetaData rsmd = rs.getMetaData(); + List resultSetColumns = new ArrayList<>(); + int columnCount = rsmd.getColumnCount(); + for (int i = 1; i <= columnCount; i++) { + Column col = new Column(); + col.setName(rsmd.getColumnName(i)); + col.setDataType(rsmd.getColumnTypeName(i)); + Column retrievedColumn = columnMap.get(col.getName()); + if (retrievedColumn != null && retrievedColumn.getReference() != null) { + col.setReference(retrievedColumn.getReference()); + } else { + logger.log(Level.WARNING, "Error setting reference on column, this can cause issues when annotations are created on the column!"); + } + resultSetColumns.add(col); + } + + List> data = new ArrayList<>(); + while (rs.next()) { + List row = new ArrayList<>(); + for (int i = 1; i <= columnCount; i++) { + row.add(rs.getObject(i)); + } + data.add(row); + } + + MaterializedDataSet result = new MaterializedDataSet(); + result.setTable(table); + result.setColumns(resultSetColumns); + result.setData(data); + return result; + } catch (SQLException exc) { + throw new DataRetrievalException(exc); + } finally { + try { + if (rs != null) { + rs.close(); + } + jdbcRetrievalResult.close(); + } catch (SQLException exc) { + throw new DataRetrievalException(exc); + } + } + + } + + @Override + public JDBCRetrievalResult retrieveTableAsJDBCResultSet(Table table) { + JDBCConnectionInfo connectionInfo = (JDBCConnectionInfo) this.metaDataStore.getConnectionInfo(table); + Connection connection = null; + PreparedStatement stat = null; + try { + connection = this.getJDBCConnection(connectionInfo); + if (connection == null) { + logger.log(Level.FINE, "No jdbc connection found for table ''{0}'' (''{1}'')", new Object[]{table.getName(), table.getReference().getUrl()}); + return null; + } + String schemaName = connectionInfo.getSchemaName(); + String sql = "select * from " + quoteForJDBC(schemaName) + "." + quoteForJDBC(table.getName()); + logger.log(Level.FINER, "Running JDBC statement: ''{0}''", sql); + stat = connection.prepareStatement(sql); + return new JDBCRetrievalResult(connection, stat); + } catch (SQLException exc) { + String msg = MessageFormat.format("An SQL exception occurred when preparing data access for table ''{0}'' ({1})", new Object[]{table.getName(), table.getReference().getUrl()}); + logger.log(Level.WARNING, msg, exc); + try { + if (connection != null) { + connection.close(); + } + } catch (SQLException exc2) { + // do nothing + logger.log(Level.WARNING, msg, exc2); + throw new DataRetrievalException(exc2); + } + throw new DataRetrievalException(exc); + } + } +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/JDBCRetrievalResult.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/JDBCRetrievalResult.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/JDBCRetrievalResult.java new file mode 100755 index 0000000..b418eb0 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/JDBCRetrievalResult.java @@ -0,0 +1,48 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.connectivity; + +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.SQLException; + +public class JDBCRetrievalResult { + + private Connection connection; + private PreparedStatement preparedStatement; + + public JDBCRetrievalResult(Connection connection, PreparedStatement preparedStatement) { + super(); + this.connection = connection; + this.preparedStatement = preparedStatement; + } + + public Connection getConnection() { + return connection; + } + + public PreparedStatement getPreparedStatement() { + return preparedStatement; + } + + public void close() throws SQLException { + if (preparedStatement != null) { + preparedStatement.close(); + } + if (connection != null) { + connection.close(); + } + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/RESTClientManager.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/RESTClientManager.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/RESTClientManager.java new file mode 100755 index 0000000..7946084 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/connectivity/RESTClientManager.java @@ -0,0 +1,88 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.connectivity; + +import java.net.URI; +import java.security.GeneralSecurityException; +import java.security.cert.X509Certificate; +import java.util.logging.Logger; + +import javax.net.ssl.SSLContext; + +import org.apache.http.auth.UsernamePasswordCredentials; +import org.apache.http.client.HttpClient; +import org.apache.http.client.fluent.Executor; +import org.apache.http.conn.ssl.NoopHostnameVerifier; +import org.apache.http.conn.ssl.SSLConnectionSocketFactory; +import org.apache.http.impl.client.HttpClientBuilder; +import org.apache.http.ssl.SSLContextBuilder; +import org.apache.http.ssl.TrustStrategy; + +/** + * + * This is a helper class to authenticate http requests + * + */ +public class RESTClientManager { + + Logger logger = Logger.getLogger(RESTClientManager.class.getName()); + + private Executor executor = null; + + private URI baseUrl; + private String user; + private String password; + + public RESTClientManager(URI baseUrl, String user, String password) { + this.baseUrl = baseUrl; + this.user = user; + this.password = password; + } + + public RESTClientManager(URI baseUrl) { + this(baseUrl, null, null); + } + + public Executor getAuthenticatedExecutor() throws GeneralSecurityException { + if (executor != null) { + return executor; + } + // TODO always accept the certificate for now but do proper certificate stuff in the future + TrustStrategy acceptAllTrustStrategy = new TrustStrategy() { + @Override + public boolean isTrusted(X509Certificate[] certificate, String authType) { + return true; + } + }; + SSLContextBuilder contextBuilder = new SSLContextBuilder(); + SSLContext context = contextBuilder.loadTrustMaterial(null, acceptAllTrustStrategy).build(); + SSLConnectionSocketFactory scsf = new SSLConnectionSocketFactory(context, new NoopHostnameVerifier()); + + HttpClient httpClient = HttpClientBuilder.create() // + .setSSLSocketFactory(scsf) // + .build(); + + if (this.user != null) { + if (this.baseUrl == null) { + executor = Executor.newInstance(httpClient).auth(new UsernamePasswordCredentials(this.user, this.password)); + } else { + executor = Executor.newInstance(httpClient).auth(this.baseUrl.getHost(), new UsernamePasswordCredentials(this.user, this.password)); + } + } else { + executor = Executor.newInstance(httpClient); + } + return executor; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/AnalysisRequestTracker.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/AnalysisRequestTracker.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/AnalysisRequestTracker.java new file mode 100755 index 0000000..23ef661 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/AnalysisRequestTracker.java @@ -0,0 +1,130 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.discoveryservice; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.atlas.odf.api.analysis.AnalysisRequest; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; +import org.apache.atlas.odf.api.analysis.AnalysisRequestTrackerStatus; + +// JSON +@ApiModel(description="Container for tracking the status of an analysis request.") +public class AnalysisRequestTracker { + + @ApiModelProperty(value="Analysis request", required=true) + private AnalysisRequest request; + + @ApiModelProperty(value="List of discovery service requests that make up the analysis request", required=true) + private List discoveryServiceRequests = new ArrayList(); + + @ApiModelProperty(value="List of responses, one for each discovery service request", required=true) + private List discoveryServiceResponses = new ArrayList(); + + @ApiModelProperty(value="Status of the analysis request", required=true) + private AnalysisRequestTrackerStatus.STATUS status = AnalysisRequestTrackerStatus.STATUS.INITIALIZED; + + @ApiModelProperty(value="Detailed status of the analysis request", required=false) + private String statusDetails; + + @ApiModelProperty(value="Timestamp of last status update", required=true) + private long lastModified; + + @ApiModelProperty(value="User who has submitted the analysis request", required=true) + private String user; + + // A tracker object is used to publish changes across all ODF nodes. When writing a tracker on the queue, + // a revision is added so that we know when a tracker has successfully been stored in the ODF that wrote it. + // This is necessary to make storing of these trackers a synchronous method. + @ApiModelProperty(value="Internal revision id of the analysis request", required=true) + private String revisionId; + + @ApiModelProperty(value="Next discovery service request to be issued") + private int nextDiscoveryServiceRequest = 0; + + public String getUser() { + return user; + } + + public void setUser(String user) { + this.user = user; + } + + public long getLastModified() { + return lastModified; + } + + public void setLastModified(long lastModified) { + this.lastModified = lastModified; + } + + public List getDiscoveryServiceRequests() { + return discoveryServiceRequests; + } + + public void setDiscoveryServiceRequests(List discoveryServiceRequests) { + this.discoveryServiceRequests = discoveryServiceRequests; + } + + public int getNextDiscoveryServiceRequest() { + return nextDiscoveryServiceRequest; + } + + public void setNextDiscoveryServiceRequest(int nextDiscoveryServiceRequest) { + this.nextDiscoveryServiceRequest = nextDiscoveryServiceRequest; + } + + public AnalysisRequest getRequest() { + return request; + } + + public void setRequest(AnalysisRequest request) { + this.request = request; + } + + public AnalysisRequestTrackerStatus.STATUS getStatus() { + return status; + } + + public void setStatus(AnalysisRequestTrackerStatus.STATUS status) { + this.status = status; + } + + public String getStatusDetails() { + return statusDetails; + } + + public void setStatusDetails(String statusDetails) { + this.statusDetails = statusDetails; + } + + public List getDiscoveryServiceResponses() { + return discoveryServiceResponses; + } + + public void setDiscoveryServiceResponses(List discoveryServiceResponses) { + this.discoveryServiceResponses = discoveryServiceResponses; + } + + public String getRevisionId() { + return revisionId; + } + + public void setRevisionId(String revisionId) { + this.revisionId = revisionId; + } +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DataSetCheckResult.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DataSetCheckResult.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DataSetCheckResult.java new file mode 100755 index 0000000..3e46e83 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DataSetCheckResult.java @@ -0,0 +1,54 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.discoveryservice; + +import io.swagger.annotations.ApiModel; +import io.swagger.annotations.ApiModelProperty; + +/** + * + * An object of this class must be returned by a services checkDataSet method. + * + */ +@ApiModel(description="Result returned by REST-based discovery services that indicates whether a dataset can be processed by the service.") +public class DataSetCheckResult { + + public static enum DataAccess { + NotPossible, + Possible + }; + + @ApiModelProperty(value="Indicates whether a dataset can be accessed by a discovery service, i.e. whether access is possible or not", readOnly=true, required=true) + private DataAccess dataAccess = DataAccess.Possible; + + @ApiModelProperty(value="Message explaining why access to the dataset is not possible", readOnly=true) + private String details; + + public DataAccess getDataAccess() { + return dataAccess; + } + + public void setDataAccess(DataAccess dataAccess) { + this.dataAccess = dataAccess; + } + + public String getDetails() { + return details; + } + + public void setDetails(String details) { + this.details = details; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryService.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryService.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryService.java new file mode 100755 index 0000000..99366e7 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryService.java @@ -0,0 +1,42 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.discoveryservice; + +import java.util.concurrent.ExecutorService; + +import org.apache.atlas.odf.api.metadata.MetadataStore; +import org.apache.atlas.odf.api.annotation.AnnotationStore; +import org.apache.atlas.odf.api.discoveryservice.datasets.DataSetContainer; + +/** + * Every kind of discovery service must implement this interface + * For java services, the executor service can be used to start / manage threads with credentials of the current ODF user + * The metadata store can be used to access metadata required by the service. + * + */ +public interface DiscoveryService { + + void setExecutorService(ExecutorService executorService); + + void setMetadataStore(MetadataStore metadataStore); + void setAnnotationStore(AnnotationStore annotationStore); + + /** + * Checks whether a data set can be processed by the discovery service. + * + * @param dataSetContainer Data set container that contains a reference to the data set to be accessed + * @return Status information whether access to the data set is possible or not + */ + DataSetCheckResult checkDataSet(DataSetContainer dataSetContainer); +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceBase.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceBase.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceBase.java new file mode 100755 index 0000000..db73966 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceBase.java @@ -0,0 +1,54 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.discoveryservice; + +import java.util.concurrent.ExecutorService; + +import org.apache.atlas.odf.api.metadata.MetadataStore; +import org.apache.atlas.odf.api.annotation.AnnotationStore; +import org.apache.atlas.odf.api.discoveryservice.datasets.DataSetContainer; + +/** + * A discovery service base class that services can subclass for convenience. + * + * + */ +public abstract class DiscoveryServiceBase implements DiscoveryService { + protected ExecutorService executorService; + protected MetadataStore metadataStore; + protected AnnotationStore annotationStore; + + @Override + public void setExecutorService(ExecutorService executorService) { + this.executorService = executorService; + } + + @Override + public void setMetadataStore(MetadataStore metadataStore) { + this.metadataStore = metadataStore; + } + + @Override + public void setAnnotationStore(AnnotationStore annotationStore) { + this.annotationStore = annotationStore; + } + + @Override + public DataSetCheckResult checkDataSet(DataSetContainer dataSet) { + DataSetCheckResult result = new DataSetCheckResult(); + result.setDataAccess(DataSetCheckResult.DataAccess.Possible); + return result; + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceEndpoint.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceEndpoint.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceEndpoint.java new file mode 100755 index 0000000..d3f63e8 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceEndpoint.java @@ -0,0 +1,50 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.discoveryservice; + +import java.util.HashMap; +import java.util.Map; + +import org.apache.wink.json4j.JSONException; + +import com.fasterxml.jackson.annotation.JsonAnyGetter; +import com.fasterxml.jackson.annotation.JsonAnySetter; + +import io.swagger.annotations.ApiModel; + +//JSON +@ApiModel(description="Endpoint of the discovery service.") +public class DiscoveryServiceEndpoint { + private String runtimeName; + private Map props = new HashMap<>(); + + public String getRuntimeName() { + return runtimeName; + } + + public void setRuntimeName(String runtimeName) { + this.runtimeName = runtimeName; + } + + @JsonAnyGetter + public Map get() { + return props; + } + + @JsonAnySetter + public void set(String name, Object value) { + props.put(name, value); + } + +} http://git-wip-us.apache.org/repos/asf/incubator-atlas/blob/6d19e129/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceJavaEndpoint.java ---------------------------------------------------------------------- diff --git a/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceJavaEndpoint.java b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceJavaEndpoint.java new file mode 100755 index 0000000..8e79511 --- /dev/null +++ b/odf/odf-api/src/main/java/org/apache/atlas/odf/api/discoveryservice/DiscoveryServiceJavaEndpoint.java @@ -0,0 +1,50 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.atlas.odf.api.discoveryservice; + +/** + * + * This class represents a java ODF discovery service endpoint. + * Note: It doesn't inherit from DiscoveryServiceEndpoint. To convert this from / to this class use JSONUtils.convert() + * + */ +public class DiscoveryServiceJavaEndpoint { + + private String runtimeName; + /* + * The class name identifies a class that must be available on the classpath and implements the ODF service interface + */ + private String className; + + public DiscoveryServiceJavaEndpoint() { + this.setRuntimeName("Java"); + } + + public String getClassName() { + return className; + } + + public void setClassName(String className) { + this.className = className; + } + + public String getRuntimeName() { + return runtimeName; + } + + public void setRuntimeName(String runtimeName) { + this.runtimeName = runtimeName; + } + +}