nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumanth Chinthagunta <xmlk...@gmail.com>
Subject Re: DB2 to MS SQL data transfer
Date Sat, 13 Aug 2016 01:01:18 GMT
Thanks Matt,
I already have scheme replicated. Will try this method and let you know.
If first time transfer works, with QueryDatabaseTable I should be able to replicate new records
automatically, which is what I am planing to have.
Thanks,
-Sumo

Sent from my iPhone

> On Aug 12, 2016, at 8:46 AM, Matt Burgess <mattyb149@gmail.com> wrote:
> 
> Sumo,
> 
> One-time migrations are possible with NiFi (although probably not a
> common use case). Are the target tables created already? Some DBs
> (like MySQL) support queries that return the Create Table statement,
> if you get those you can use PutSQL to execute them on your target
> system.
> 
> If the target tables are created, you can use QueryDatabaseTable ->
> ConvertAvroToJSON -> ConvertJsonToSQL -> PutSQL, the conversions
> should leave you with flowfiles containing INSERT statements. If you
> already know the schema, you could probably use ReplaceText instead of
> ConvertJsonToSQL.
> 
> QueryDatabaseTable [1], if you configure it with a maximum-value
> column like the primary key column, will only fetch the data once. It
> keeps track of the largest value in the column you specify, so as long
> as more data (with higher values in that column) are not added to the
> source table, then QueryDatabaseTable can continue to "run" but it
> won't output any records after the first full fetch.
> 
> Another option might be to export the source table to a file, then
> transfer it with NiFi to the target system, then use PutSQL to do a
> BULK INSERT [2].
> 
> Regards,
> Matt
> 
> [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html
> [2] https://msdn.microsoft.com/en-us/library/ms188365.aspx
> 
>> On Thu, Aug 11, 2016 at 7:09 PM, Sumanth Chinthagunta <xmlking@gmail.com> wrote:
>> I have A set of tables to transfer, but as I have small set of tables , I can duplicate
flows if flow design works with one table.
>> Basically migrating an app from mainframe/DB2 to MS SQL server.
>> Thanks
>> Sumo
>> Sent from my iPhone
>> 
>>> On Aug 11, 2016, at 1:17 PM, Jeff <jtswork@gmail.com> wrote:
>>> 
>>> Hello Sumo,
>>> 
>>> Are you looking at transferring data from a specific table between two
>>> databases, or all tables?
>>> 
>>> On Wed, Aug 10, 2016 at 3:17 PM Sumanth Chinthagunta <xmlking@gmail.com>
>>> wrote:
>>> 
>>>> I have same scheme in DB2 and SQL server.
>>>> Is there a simple solution to transfer data with NiFi? ( one time )
>>>> Hoping for somebody have better  idea to build the flow :)
>>>> Thanks
>>>> Sumo
>>>> Sent from my iPhone
>>>> 

Mime
View raw message