Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Flavio Pompermaier <pompermaier@okkam.it>
Date: Wed, 6 Sep 2017 11:38:22 +0200
Message-ID: <CAELUF_B=AHOZFHn8mfiA8MaDaOd8g5B1Z=8tWtaKN8LPSuE+0w@mail.gmail.com>
Subject: Apache Phenix integration
To: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="001a1143f4700624690558821c33"
archived-at: Wed, 06 Sep 2017 09:38:47 -0000

--001a1143f4700624690558821c33
Content-Type: text/plain; charset="UTF-8"

Hi to all,
I'm writing a job that uses Apache Phoenix.

At first I used the PhoenixOutputFormat as (hadoop) OutputFormat but it's
not well suited to work with Table API because it cannot handle generic
objects like Rows (it need a DBWritable Object that should be already
present at compile time). So I've looked into the code of the
PhoenixOutputFormat and it's nothing else than a JDBCOutputFormat
(basically).

However, to make it work I had to slightly modify the Flink
JDBCOutputformat, adding a dbConn.commit() after the executeBatch() on the
PreparedStatement. E.g:

    upload.executeBatch();
    dbConn.commit();

For the moment I've just created a dedicated PhoenixJdbcOutpuFormat where
I've added these 2 lines of code starting from the code of the
JDBCOutputformat (it couldn't be extended in this case because all fields
are private).

What do you think about this? Should I open a ticket to add a connection
commit after executeBatch (in order to be compatible with Phoenix) or
something else (e.g. create a Phoenix connector that basically extend
JDBCOutputConnector and ovewrite 2 methods, changing also the visibility of
its fields to protected)?

Best,
Flavio

--001a1143f4700624690558821c33
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi to all,<div>I&#39;m writing a job that uses Apache Phoe=
nix.</div><div><br></div><div>At first I used the PhoenixOutputFormat as (h=
adoop) OutputFormat but it&#39;s not well suited to work with Table API bec=
ause it cannot handle generic objects like Rows (it need a DBWritable Objec=
t that should be already present at compile time). So I&#39;ve looked into =
the code of the PhoenixOutputFormat and it&#39;s nothing else than a JDBCOu=
tputFormat (basically).</div><div><br></div><div>However, to make it work I=
 had to slightly modify the Flink JDBCOutputformat, adding a dbConn.commit(=
) after the executeBatch() on the PreparedStatement. E.g:</div><div><br></d=
iv><div>=C2=A0 =C2=A0 upload.executeBatch();</div><div>=C2=A0 =C2=A0 dbConn=
.commit();</div><div><br></div><div>For the moment I&#39;ve just created a =
dedicated PhoenixJdbcOutpuFormat where I&#39;ve added these 2 lines of code=
 starting from the code of the JDBCOutputformat (it couldn&#39;t be extende=
d in this case because all fields are private).</div><div><br></div><div>Wh=
at do you think about this? Should I open a ticket to add a connection comm=
it after executeBatch (in order to be compatible with Phoenix) or something=
 else (e.g. create a Phoenix connector that basically extend JDBCOutputConn=
ector and ovewrite 2 methods, changing also the visibility of its fields to=
 protected)?</div><div><br></div><div>Best,</div><div>Flavio<br><br>
</div></div>

--001a1143f4700624690558821c33--