From dev-return-23260-archive-asf-public=cust-asf.ponee.io@atlas.apache.org Mon May 7 23:30:34 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 63050180648 for ; Mon, 7 May 2018 23:30:34 +0200 (CEST) Received: (qmail 81212 invoked by uid 500); 7 May 2018 21:30:33 -0000 Mailing-List: contact dev-help@atlas.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.apache.org Delivered-To: mailing list dev@atlas.apache.org Received: (qmail 81197 invoked by uid 99); 7 May 2018 21:30:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 May 2018 21:30:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 66E3F180820; Mon, 7 May 2018 21:30:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.439 X-Spam-Level: * X-Spam-Status: No, score=1.439 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_MED=-2.3, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id EsIDWmnXsHrC; Mon, 7 May 2018 21:30:30 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 439105F1E7; Mon, 7 May 2018 21:30:29 +0000 (UTC) Received: from reviews.apache.org (unknown [10.41.0.12]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id E1FEDE00CC; Mon, 7 May 2018 21:30:28 +0000 (UTC) Received: from reviews-vm2.apache.org (localhost [IPv6:::1]) by reviews.apache.org (ASF Mail Server at reviews-vm2.apache.org) with ESMTP id A60B2C4021A; Mon, 7 May 2018 21:30:28 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============8603873280901038293==" MIME-Version: 1.0 Subject: Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0 From: Ashutosh Mestry To: Madhan Neethiraj , David Radley , Sarath Subramanian Cc: Ashutosh Mestry , atlas , Pierre Padovani Date: Mon, 07 May 2018 21:30:28 -0000 Message-ID: <20180507213028.3181.83378@reviews-vm2.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Ashutosh Mestry X-ReviewGroup: atlas X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/66064/ X-Sender: Ashutosh Mestry X-ReviewBoard-ShipIt: 1 References: <20180507212815.3181.93068@reviews-vm2.apache.org> In-Reply-To: <20180507212815.3181.93068@reviews-vm2.apache.org> X-ReviewBoard-ShipIt-Only: 1 Reply-To: Ashutosh Mestry X-ReviewRequest-Repository: atlas --===============8603873280901038293== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66064/#review202597 ----------------------------------------------------------- Ship it! Ship It! - Ashutosh Mestry On May 7, 2018, 9:28 p.m., Pierre Padovani wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66064/ > ----------------------------------------------------------- > > (Updated May 7, 2018, 9:28 p.m.) > > > Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian. > > > Bugs: ATLAS-2478 > https://issues.apache.org/jira/browse/ATLAS-2478 > > > Repository: atlas > > > Description > ------- > > This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation. > > Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well. > > NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed. > > **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 > distro/pom.xml 6431fd86d > distro/src/bin/atlas_config.py 9062da649 > distro/src/bin/atlas_start.py 61d69eb21 > distro/src/bin/atlas_stop.py 94c3d6d46 > distro/src/conf/atlas-env.sh 053cbd500 > distro/src/main/assemblies/standalone-package.xml dc88add56 > docs/src/site/twiki/Configuration.twiki 63c3fce96 > docs/src/site/twiki/HighAvailability.twiki 4270d0974 > docs/src/site/twiki/InstallationSteps.twiki dca0618e3 > graphdb/janus/pom.xml 016a09c33 > pom.xml 3469a393a > webapp/pom.xml 03b84087f > > > Diff: https://reviews.apache.org/r/66064/diff/3/ > > > Testing > ------- > > We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly. > > > Thanks, > > Pierre Padovani > > --===============8603873280901038293==--