nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Quinn <Bryan.Qu...@asavie.com>
Subject RE: Issue with GenerateTableFetch bulk load when using MS SQL Server database type
Date Mon, 24 Jul 2017 13:55:09 GMT
Hi Matt.
I’ll have a look at option a. What’s the timeframe for a fix and prod release for this
assuming it’s a bug? Ballpark.

Thanks,
Bryan

From: Matt Burgess [mailto:mattyb149@gmail.com]
Sent: Monday 24 July 2017 14:40
To: users@nifi.apache.org
Subject: Re: Issue with GenerateTableFetch bulk load when using MS SQL Server database type

Bryan,

This is likely a bug, I will investigate and write it up if so. In the meantime, are you planning
on doing the bulk fetch in parallel (with a Remote Process Group into ExecuteSQL across a
NiFi cluster)? If not, you may find that QueryDatabaseTable is a good alternative, it can
be configured to fetch N rows at a time. The trade off is that it is not parallel or concurrent
(each N rows will be fetched one batch at a time), but then each batch can be processed concurrently
or in parallel.

Another alternative, if you know the approximate size of the database, is to use GenerateFlowFile
to write the SQL statements yourself, then SplitText to get one statement per flow file, then
ExecuteSQL (concurrently or in parallel). This solution does not scale with your database
size but with some copy and paste (and updating as the DB grows), can emulate GenerateTableFetch
in the short term.

Regards,
Matt

On Jul 24, 2017, at 8:55 AM, Bryan Quinn <Bryan.Quinn@asavie.com<mailto:Bryan.Quinn@asavie.com>>
wrote:
Hi,
I’m trying to use the GenerateTableFetch component for ‘MS SQL 2012+’ database – for
a full table export each time its run.
However I’m forced to set a column name in the ‘Maximum-value Columns’ field when the
component is run (error only shows during execution if I omit this setting).
I’m seeing the following error:
IllegalArgumentException: Order by clause cannot be null or empty when using row paging

So it looks like it only works with incremental mode – which I don’t want.

Can someone see why this is the case or is it a bug?
I’m not getting this error if I set the database type to ‘Generic’ – but unfortunately
the sql it generates isn’t valid for MS SQL server.
I’m using Nifi v1.3.0

Thanks,
Bryan

---
Bryan Quinn | Asavie
Principal Analytics Architect
E: bryan.quinn@asavie.com<mailto:bryan.quinn@asavie.com>
W: www.asavie.com<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Foutlook.office.com%2Fowa%2Fwww.asavie.com&data=02%7C01%7CBryan.Quinn%40asavie.com%7C618b8582903b4775a3f108d4d299870d%7Cff38a05d2e5248389b077fbd50bbae9f%7C0%7C0%7C636365004244466091&sdata=02nM2H4yXmxB4rpeCQGkQaUSec18VB%2FJqoffqekjp74%3D&reserved=0>
| T: @asavie

Mime
View raw message