atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Padovani <pie...@padovani.org>
Subject Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0
Date Tue, 08 May 2018 15:00:54 GMT


> On May 7, 2018, 11:35 p.m., Ashutosh Mestry wrote:
> > graphdb/janus/pom.xml
> > Lines 121 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017713#file2017713line121>
> >
> >     This is getting included even when the profile is not selected.

Per the comment about elasticsearch jars in the main distribution. 

The new version is safer now after Elastic removed all of the shaded jars... and smaller.

We either keep them, or force them to download and copy it into their distribution during
setup/installation.


- Pierre


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202602
-----------------------------------------------------------


On May 8, 2018, 3 p.m., Pierre Padovani wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
> 
> (Updated May 8, 2018, 3 p.m.)
> 
> 
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2478
>     https://issues.apache.org/jira/browse/ATLAS-2478
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
> 
> Included with this patch is an update to the berkley-elasticsearch profile to automatically
download and include elasticsearch as a side application much like solr is. Updates to the
start/stop/conf scripts are included as well.
> 
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java.
There are six constants that are incorrectly named with a '.' (dot). This is not supported
in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively
thought of as an object. In the case of the fields defined in the Constants.java file, 'type'
is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch
sees this as an error, since it cannot convert type to an object. The fix included simply
changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect
compatibility with hbase/solr for new installs. For existing installations, a reindex will
be required as the field names will have changed.
> 
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph
store by using a maven plugin that will download and run an elasticsearch node. This is nothing
more than a maven change, and change to the atlas-application.properties to switch to elasticsearch
from solr. I did not implement this, but am curious if this change would be desired. If so,
this can be done with a separate ticket.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15 
>   distro/pom.xml 6431fd86d 
>   distro/src/bin/atlas_config.py 9062da649 
>   distro/src/bin/atlas_start.py 61d69eb21 
>   distro/src/bin/atlas_stop.py 94c3d6d46 
>   distro/src/conf/atlas-env.sh 053cbd500 
>   distro/src/main/assemblies/standalone-package.xml dc88add56 
>   docs/src/site/twiki/Configuration.twiki 63c3fce96 
>   docs/src/site/twiki/HighAvailability.twiki 4270d0974 
>   docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
>   graphdb/janus/pom.xml 016a09c33 
>   pom.xml 3469a393a 
>   webapp/pom.xml 03b84087f 
> 
> 
> Diff: https://reviews.apache.org/r/66064/diff/4/
> 
> 
> Testing
> -------
> 
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have
found no issues with it. Additionally, with the update to the berkley-elasticsearch profile,
I have extensively tested that profile to make sure that management of Elasticsearch functioned
correctly.
> 
> 
> Thanks,
> 
> Pierre Padovani
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message