falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <anuj.o.ku...@accenture.com>
Subject Need help in understanding Lineage With Falcon
Date Tue, 14 Jul 2015 22:51:46 GMT
Hi,
 I am working with a client that uses Informatica Metadata Manager to visualise Lineage Information.
Informatica Metadata Manager is currently used at Data Warehouse layer and is proven effective.
But unfortunately Informatica Metadata Manage does not have any connectors to Hadoop to collect
metadata information, which makes it not so desirable tool for the entire end to end chain.
This is where Apache Falcon comes to the rescue.

Looking at Falcon, I see that Falcon exposes a set of REST APIs that can be used to capture
metadata information about process,feed and cluster entities (assuming that the workflow is
scheduled using Apache Falcon). So we are exploring option on how we can actually generate
metadata at Hadoop layer that can then be used to feed informatica Metadata Manager, which
will combine it with its own metadata from DWH and Business reports to provide a complete
Lineage information.

I have three specific question with regard to the above problem :


  1.  Where is the Metadata Repository located for Apache Falcon? Is it the config store on
Hadoop or Hcatalog ?
  2.  Is there a way to connect to this repository(for e.g.. via JDBC) ?
  3.  What set of REST APIs can be called from outside of the Falcon environment to capture
the Metadata Information about the processes scheduled using Falcon ? I looked at these<http://falcon.apache.org/0.6.1/restapi/>
set of REST APIs, which was a start for me, but I got lost in the details.

Your quick answer would be really appreciated.

Thanks,
Anuj Kumar
Technology Architect - Emerging Technology Innovation group
mobile: +31 6 30458915
ITO Toren - Gustav Mahlerplein 90 - 1082MA Amsterdam
             >
accenture

________________________________

This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise confidential information. If you have received it in error, please notify the
sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
Where allowed by local law, electronic communications with Accenture and its affiliates, including
e-mail and instant messaging (including content), may be scanned by our systems for the purposes
of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message