Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
Date: Tue, 14 Feb 2017 23:43:41 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@nifi.apache.org
Message-ID: <JIRA.13031253.1483093189000.81660.1487115821901@Atlassian.JIRA>
In-Reply-To: <JIRA.13031253.1483093189000@Atlassian.JIRA>
References: <JIRA.13031253.1483093189000@Atlassian.JIRA> <JIRA.13031253.1483093189387@jira-lw-us.apache.org>
Subject: [jira] [Commented] (NIFI-3268) Add AUTO_INCREMENT column in
 GenerateTableFetch to benefit index
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 14 Feb 2017 23:43:47 -0000


    [ https://issues.apache.org/jira/browse/NIFI-3268?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15866=
946#comment-15866946 ]=20

ASF GitHub Bot commented on NIFI-3268:
--------------------------------------

Github user qfdk commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/1376#discussion_r101172664
 =20
    --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processor=
s/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java=
 ---
    @@ -223,19 +237,34 @@ public void onTrigger(final ProcessContext contex=
t, final ProcessSessionFactory
                 }
                 final int numberOfFetches =3D (partitionSize =3D=3D 0) ? r=
owCount : (rowCount / partitionSize) + (rowCount % partitionSize =3D=3D 0 ?=
 0 : 1);
    =20
    +            if("null".equals(indexValue)) {
    +                // Generate SQL statements to read "pages" of data
    +                for (int i =3D 0; i < numberOfFetches; i++) {
    +                    FlowFile sqlFlowFile;
    =20
    -            // Generate SQL statements to read "pages" of data
    -            for (int i =3D 0; i < numberOfFetches; i++) {
    -                FlowFile sqlFlowFile;
    +                    Integer limit =3D partitionSize =3D=3D 0 ? null : =
partitionSize;
    +                    Integer offset =3D partitionSize =3D=3D 0 ? null :=
 i * partitionSize;
    +                    final String query =3D dbAdapter.getSelectStatemen=
t(tableName, columnNames, whereClause, StringUtils.join(maxValueColumnNameL=
ist, ", "), limit, offset);
    +                    sqlFlowFile =3D session.create();
    +                    sqlFlowFile =3D session.write(sqlFlowFile, out -> =
{
    +                        out.write(query.getBytes());
    +                    });
    +                    session.transfer(sqlFlowFile, REL_SUCCESS);
    +                }
    +            }else {
    +                for (int i =3D 0; i < numberOfFetches; i++) {
    +                    FlowFile sqlFlowFile;
    =20
    -                Integer limit =3D partitionSize =3D=3D 0 ? null : part=
itionSize;
    -                Integer offset =3D partitionSize =3D=3D 0 ? null : i *=
 partitionSize;
    -                final String query =3D dbAdapter.getSelectStatement(ta=
bleName, columnNames, whereClause, StringUtils.join(maxValueColumnNameList,=
 ", "), limit, offset);
    -                sqlFlowFile =3D session.create();
    -                sqlFlowFile =3D session.write(sqlFlowFile, out -> {
    -                    out.write(query.getBytes());
    -                });
    -                session.transfer(sqlFlowFile, REL_SUCCESS);
    +                    Integer limit =3D partitionSize;
    +                    whereClause =3D indexValue + " >=3D " + limit * i;
    --- End diff --
   =20
    Thank you for your advice. I will work on it and solve the conflit.


> Add AUTO_INCREMENT column in GenerateTableFetch to benefit index
> ----------------------------------------------------------------
>
>                 Key: NIFI-3268
>                 URL: https://issues.apache.org/jira/browse/NIFI-3268
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.1.1
>         Environment: - ubuntu 16.04
> - java version "1.8.0_111"
> - Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> - Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
>            Reporter: qfdk
>              Labels: easyfix
>             Fix For: 1.2.0
>
>
> I added AUTO_INCREMENT column in  GenerateTableFetch to benefit index col=
umn
> By default this processor uses OFFSET, i have  problems with large data. =
somme column has index so we could use index to speed up query time.
> I posted question here :
> https://community.hortonworks.com/questions/72586/how-can-i-use-an-array-=
with-putelasticsearch.html
> If you indexed un column (id), you could use this sql
> ```
> select xxx
> From xxxxx
> where 200000=3D>id
> order by id
> limit 200000
> ```
> =E2=80=9COFFSET is bad for skipping previous rows.=E2=80=9D [Online]. Ava=
ilable: http://Use-The-Index-Luke.com/sql/partial-results/fetch-next-page. =
[Accessed: 27-Dec-2016].
> Thank you in advance


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)