ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 28197: Falcon lineage needs changes to startup.properties
Date Wed, 19 Nov 2014 01:36:54 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28197/
-----------------------------------------------------------

(Updated Nov. 19, 2014, 1:36 a.m.)


Review request for Ambari, Jaimin Jetly, Mahadev Konar, Srimanth Gunturi, and Sid Wagle.


Bugs: AMBARI-8379
    https://issues.apache.org/jira/browse/AMBARI-8379


Repository: ambari


Description
-------

Summary of changes needed and implications

changes in falcon/conf/startup.properties:

1) For Falcon lineage to work below changes are needed on Ambari deployed configs for Falcon
Implications : Lineage feature of Falcon is blocked without these changes

MetadataMappingService is required for lineage:
```
*.application.services=org.apache.falcon.security.AuthenticationInitializationService,\
                        org.apache.falcon.workflow.WorkflowJobEndNotificationService,\
                        org.apache.falcon.service.ProcessSubscriberService,\
                        org.apache.falcon.entity.store.ConfigurationStore,\
                        org.apache.falcon.rerun.service.RetryService,\
			org.apache.falcon.rerun.service.LateRunService,\
		        org.apache.falcon.service.LogCleanupService,\
                        org.apache.falcon.metadata.MetadataMappingService
```

```
######### Graph Database Properties #########
# Graph implementation
*.falcon.graph.blueprints.graph=com.thinkaurelius.titan.core.TitanFactory

# Graph Storage
*.falcon.graph.storage.directory=/grid/0/data/lineage/graphdb
*.falcon.graph.storage.backend=berkeleyje
*.falcon.graph.serialize.path=/grid/0/data/lineage
*.falcon.graph.preserve.history=false
```

The properties that need a directory in the local file system should detect a mount point
that is available on the Falcon server, and the python scripts need to crease these directories
when the Falcon service is configured.

The folder permission are,
owner:falcon user
group:hadoop
permissions:775


Diffs (updated)
-----

  ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py
37dd213 
  ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/params.py
2316326 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/configuration/falcon-startup.properties.xml
ec8b9c7 
  ambari-web/app/data/HDP2/site_properties.js e9b8234 
  ambari-web/app/models/service_config.js 255a679 

Diff: https://reviews.apache.org/r/28197/diff/


Testing
-------

Ran ambari-server unit tests, all of which passed.
OK
----------------------------------------------------------------------
Total run:664
Total errors:0
Total failures:0
OK

Also used the Install Wizard to deploy an HDP 2.2 cluster with 2 nodes, one of which was the
Falcon server, and the other the client.
The falcon-startup.properties contains the correct values, and the Falcon Service Check passed.

http://c6411.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=falcon-startup.properties&tag=version1


* *.application.services: "org.apache.falcon.security.AuthenticationInitializationService,\
org.apache.falcon.workflow.WorkflowJobEndNotificationService, \ org.apache.falcon.service.ProcessSubscriberService,\
org.apache.falcon.entity.store.ConfigurationStore,\ org.apache.falcon.rerun.service.RetryService,\
org.apache.falcon.rerun.service.LateRunService,\ org.apache.falcon.service.LogCleanupService,\
org.apache.falcon.metadata.MetadataMappingService",
* *.falcon.graph.blueprints.graph: "com.thinkaurelius.titan.core.TitanFactory",
* *.falcon.graph.preserve.history: "false",
* *.falcon.graph.serialize.path: "/hadoop/falcon/data/lineage",
* *.falcon.graph.storage.backend: "berkeleyje",
* *.falcon.graph.storage.directory: "/hadoop/falcon/data/lineage/graphdb",


Further, the permissions are correct,
ls -la /hadoop/falcon/data/
drwxrwxr-x 3 falcon hadoop 4096 Nov 18 23:54 lineage
ls -la /hadoop/falcon/data/lineage/
drwxrwxr-x 2 falcon hadoop 4096 Nov 19 00:03 graphdb


Thanks,

Alejandro Fernandez


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message