nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Ball <>
Subject Re: Sqoop Support in NIFI
Date Tue, 29 Mar 2016 09:32:39 GMT
Another processor that may be of interest to you is the QueryDatabaseTable processor, which
has just been released in 0.6.0. This provides incremental load capabilities similar to sqoop.

If you’re looking for the schema type functionality, bear in mind that the ExecuteSQL (and
new Query processor) preserve schema with Avro.

Sqoop also allows import to HBase, which you can do with PutHBaseJson (use the ConvertAvroToJson
processor to feed this).

Distributed partitoned queries isn’t in there yet, but I believe is on the way, so sqoop
may have the edge for that use case today.

Granted, NiFi doesn’t have much by way of HCatalog integration at the moment, but most of
the functionality you’ll find in Sqoop is in NiFi. Unless you are looking to move terabytes
at a time, then NiFi should be able to handle most of what you would use sqoop for, so it
would be very interesting to hear more detail on your use case, and why you needed sqoop on
top of NiFi.


On 29 Mar 2016, at 09:06, prabhu Mahendran <<>>


Yes, In my case i have created the Custom processor with Sqoop API which accommodates complete
functionality of sqoop.
As per you concern we have able to move the data only from HDFS to SQl or Vice versa, But
sqoop having more functionality which we can achieve it by Sqoop.RunTool() in org.apache.sqoop.sqoop.
The Sqoop Java client will works well and Implement that API into new Sqoop NIFI processor
Doesn't work!

On Tue, Mar 29, 2016 at 12:49 PM, Conrad Crampton <<>>
If you could explain exactly what you are trying to achieve I.e. What part of the data pipeline
you are looking to use NiFi for and where you wish to retain Sqoop I could perhaps have a
more informed input (although I have only been using NiFi myself for a few weeks). Sqoop obviously
can move the data from RDBM systems through to HDFS (and vice versa) as can NiFi, not sure
why you would want the mix (or at least I can’t see it from the description you have provided
thus far).
I have limited knowledge of Sqoop, but either way, I am sure you could ‘drive’ Sqoop from
a custom NiFi processor if you so choose, and you can ‘drive’ NiFi externally (using the
REST api) - if Sqoop can consume it.

From: prabhu Mahendran <<>>
Reply-To: "<>" <<>>
Date: Tuesday, 29 March 2016 at 07:55
To: "<>" <<>>
Subject: Re: Sqoop Support in NIFI

Hi Conrad,

Thanks for Quick Response.

Yeah.Combination of Execute SQL and Put HDFS works well instead of Sqoop.But is there any
possible to use Sqoop(client) to do like this?

Prabhu Mahendran

On Tue, Mar 29, 2016 at 12:04 PM, Conrad Crampton <<>>
Why use sqoop at all? Use a combination of ExecuteSQL [1] and PutHDFS [2].
I have just replace the use of Flume using a combination of ListenSyslog and PutHDFS which
I guess is a similar architectural pattern.

From: prabhu Mahendran <<>>
Reply-To: "<>" <<>>
Date: Tuesday, 29 March 2016 at 07:27
To: "<>" <<>>
Subject: Sqoop Support in NIFI


I am new to nifi.

       I have to know that  "Is there is any Support for Sqoop with help of NIFI Processors?."

And in which way to done the following case with help of Sqoop.

    Move data from oracle,SqlServer,MySql into HDFS and vice versa.

Prabhu Mahendran

***This email originated outside SecureData***

Click here<> to report this email
as spam.

SecureData, combating cyber threats


The information contained in this message or any of its attachments may be privileged and
confidential and intended for the exclusive use of the intended recipient. If you are not
the intended recipient any disclosure, reproduction, distribution or other dissemination or
use of this communications is strictly prohibited. The views expressed in this email are those
of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only
valid if followed up by a formal written quote.

SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address:
SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT

View raw message