nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randy Gelhausen (JIRA)" <>
Subject [jira] [Commented] (NIFI-1575) Add ability to query DB table using "last value" from column
Date Mon, 29 Feb 2016 16:30:18 GMT


Randy Gelhausen commented on NIFI-1575:

I see use cases where the query needs to run not based exclusively on monotonically increasing
values, but on some context provided by an external trigger.

Thus a very useful (and hopefully not too complex) additional feature would be to allow the
user to pull the "last value" from DistributedMapCache. This way external flows/activities
may act as a more customizable and fine-grained control mechanism for the QueryDBTable processor.

> Add ability to query DB table using "last value" from column
> ------------------------------------------------------------
>                 Key: NIFI-1575
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
> It would be a useful feature for a processor to be able to query from a database table
for only those records that have are added/available since the last time the query was executed.
For example, if the processor could keep the max value of a timestamp column for the last
result set returned, the next time the query was issued, it could use that timestamp value
and column to only return records whose timestamps were greater than "the last time".
> This shouldn't be limited to timestamps of course; it would be useful for any strictly-increasing
value (like primary key ID) but would be up to the user to select the table and column. The
processor would simply keep the state for the specified column's max value.
> Proposed is a "QueryDBTable" processor which would have a properties to specify the table
name and the column in the table for which to keep state information about the last maximum
value retrieved. Subsequent queries of the table use this value to filter for rows whose value
for the specified column are greater than the "last maximum value". Upon each successful query,
the "last maximum value" is updated.

This message was sent by Atlassian JIRA

View raw message