nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: question about RouteOnAttribute 1.8.0 processor
Date Wed, 25 Mar 2020 19:28:34 GMT
Cody,

ExecuteSQL outputs data in Avro format and embeds the schema in the
content of the flow file (which Avro supports), leading to a non-zero
flowfile size. As of NiFi 1.8.0, you can instead use ExecuteSQLRecord
to output the data in any supported format (via a RecordSetWriter
controller service), including an AvroRecordSetWriter (if you need the
output in Avro) configured to write the schema as an attribute (or not
at all) versus embedding the schema in the content. If you are
converting the format downstream, you can instead choose a
RecordSetWriter for your desired format, and choose how the schema is
written (as an attribute or not, e.g.). Please let me know if you have
any questions, happy to help!

Regards,
Matt

On Wed, Mar 25, 2020 at 3:04 PM Cody Ferguson <cody.ferguson@gm.com> wrote:
>
> Hello,
>
> We have a process group that queries data from our database and then passes the data
on to a RouteOnAttribute processor which determines whether we actually have data to process
or not.
>
> The RouteOnAttribute processor determines that by checking the file size. If the file
size is less than so many bytes it will update a run record in the database and end, if it
is more than so many bytes it will continue on because it believes we retrieved records that
need to be processed.
>
> One thing we noticed recently in preprod is the size of an empty file was larger than
the file size we were checking against. So somehow the size of the file increased (maybe additional
meta data was added?) and because of that our check was no longer working.
>
> My question is is there a better way to determine if our ExecuteSQL 1.8.0 processor retrieved
data or not and fork the process based on that? It seems like checking the file size can lead
to issues down the road because the file itself can grow regardless of having data or not.
>
> Thanks,
>
> Cody
>
>
> Nothing in this message is intended to constitute an electronic signature unless a specific
statement to the contrary is included in this message.
>
> Confidentiality Note: This message is intended only for the person or entity to which
it is addressed. It may contain confidential and/or privileged material. Any review, transmission,
dissemination or other use, or taking of any action in reliance upon this message by persons
or entities other than the intended recipient is prohibited and may be unlawful. If you received
this message in error, please contact the sender and delete it from your computer.

Mime
View raw message