nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manik <>
Subject Re: nifi incremental fetch
Date Fri, 30 Sep 2016 04:18:34 GMT
 Hi Matt,
Thanks for your quick response!!I am  using Maximum value property as "id" but it can't work
incremental fetch could you please some idea to finesh my task 

Thanks & Regards,Manikandan Kolanjinathan
Junior Software Engineer
Pinnacle Seven Technologies 0422-4208736, 4506535 www.pinnacleseven.comDelivering your business
apps on cloud! 

      From: Matt Burgess-2 [via Apache NiFi Developer List] <>
 To: Manik <> 
 Sent: Thursday, 29 September 2016 6:03 PM
 Subject: Re: nifi incremental fetch

QueryDatabaseTable (and GenerateTableFetch) should work for
incremental fetching of rows, provided that you have at least one
column that strictly increases (like a primary key with integer ID for
example). In that case you put the column name into the processor's
Maximum Value Columns property. The first time the processor runs, it
will get all rows from the table, but then it will keep track of the
largest value it has seen for the maximum value column. The next time
it runs, it will only retrieve rows whose values in that column are
greater than the currently observed maximum.

For your example, did you select a maximum value column and include it
in the processor's Maximum Value Columns property?


On Thu, Sep 29, 2016 at 6:15 AM, Manik <[hidden email]> wrote:
> Hi Team,
> I am confused to use which processor is fit for incremental fetch in nifi.
> In nifi My task is like i need to fetch the data from sql server and doing
> Incremental fetch data and load into hadoop and i need to compare two datas
> i.e(incremantal and normal datas) merged  and  load into hive .In this task
> i already used some processor it didnt work properly .please give some about
> this task  .I already use this processor
> QueryDatabasetable--->>--SplitAvro-->>--ConvertAvroToJSON-->>--ReplaceText-->>--LogAttributes-->>--PutHdfs-->>--MergeContent-->>UpdateAttribute-->>--PutHiveStreaming
> In this processor I didnt get exact Incremental data as well as Base data
> .please guide me to solve this tasks..
> Thanks&Advance,
> Manikandan
> --
> View this message in context:
> Sent from the Apache NiFi Developer List mailing list archive at
   If you reply to this email, your message will be added to the discussion below:
  To unsubscribe from nifi incremental fetch, click here.


View this message in context:
Sent from the Apache NiFi Developer List mailing list archive at
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message