Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
Subject: Re: To get Schema for jdbc database in Flink
To: <user@flink.apache.org>
References: <e2a38a57-72e2-1438-d128-b222a8987801@ericsson.com>
 <032a37f1-8e91-c58e-eba6-f44d4d7133c9@ericsson.com>
 <CAKiyyaGwqHwfSt_RNnQzusS7M3CeJsFNUqXBavMK+HduBPW9CA@mail.gmail.com>
 <CAGr9p8D6z1Rm1+yXto8AMUSzCOFqvF1LQc_G9weGub3gHz52uQ@mail.gmail.com>
 <978b13bd-abb7-3954-710f-a48c11aaff93@ericsson.com>
 <589AD528.7000206@apache.org>
From: Punit Tandel <punit.tandel@ericsson.com>
Organization: Ericsson AB
Message-ID: <f2100a9c-66af-0422-a117-d814670d5826@ericsson.com>
Date: Wed, 8 Feb 2017 09:48:54 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Thunderbird/45.7.0
MIME-Version: 1.0
In-Reply-To: <589AD528.7000206@apache.org>
Content-Type: multipart/alternative;
	boundary="------------0C3ACB767B66BDD188E5E11C"
Received-SPF: None (protection.outlook.com: ericsson.com does not designate
 permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2017 09:48:57.5298 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0701MB2247
X-OriginatorOrg: ericsson.com
X-Brightmail-Tracker: H4sIAAAAAAAAA02SbUhTURjHO7t31+t0eJyaTxqYI8oUp5aaRWQRxL6I9SUtCx15UUk32TVJ
	Ixjqgs0yyxd0KGalOV/nG0rpopGpEIZhooRNU9EcFkImakTeHQO//Z7z/z0c/ofDUrIKsR+b
	oc7htGpVppyR0NWJfcrQiu+mxPDSbSrmUcMkdQ4pB0deokvomuRMKpeZkctpw86mSNL1xnEm
	e/3inQHdVR16Hm5ELAs4El7MpxiRhJXhdgRND9oZI3LdGYYRjC2ECAGNH1IwsPVTJAQIXwdz
	66gL2dCJoEg/7NzwwlEwYJ9EAntjfyidnkBEmhbBxNpjFyFgsAIaS+edkgc+ABbda0pgKY4F
	a02b06HxYRj6NUML7IMToW6xWUQcTxitXnCeu+JgmCyoEAtM4StQYq11MuBAGNLPioWLARsR
	bAxWikmfI7BqWhERKQ4Mbcti0j8eSr4cJX4zAzM9qy5kWGKgt9hKkwUNFJQ3IcKnwbG1gYj0
	SgTWtXGGDCUM1Bs+7VoHodfiEJHAzMD9H0W778pBz1TR7sZvD9C9rxCXoiDTnoKmPaUIx0B1
	3UdEOAD6VmsowkGgr6l1IRwCjfUO6ilimpEPz/F8VtrxEwpOm3GT5zVqhZrL6UI7n+Vtz3Zo
	P2pxnLchzCK5u9RiMCXKxKpcPi/LhoCl5N7Sk8s7R9JUVV4+p9Uka29ncrwN+bO03FcabbYn
	yHCaKoe7xXHZnPZ/KmJd/XQoia1q3Vx8k1ZlVwRqkmn3U3d1tVPRHV4l9pW5jqDPH+I9lSOR
	oyldhjVb+abH1/1h3oZDgRcikv9Yl30K17tbnliriwtmo8pka89UAfHfGgbGsjvdzF5/lb5J
	0UuFecpj6qmWG2MQ/05vidgXaoyrzE/o953r7nSLhTLNvcsJcppPV0UEU1pe9Q/pSE2bKAMA
	AA==
archived-at: Wed, 08 Feb 2017 09:49:17 -0000

--------------0C3ACB767B66BDD188E5E11C
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit

Hi Chesnay

Currently that is what i have done, reading the schema from database in 
order to create a new table in jdbc database and writing the rows coming 
from jdbcinputformat.

Overall i am trying to implement the solution which reads the streaming 
data from one source which either could be coming from kafka, Jdbc, 
Hive, Hdfs and writing those streaming data to output source which is 
again could be any of those.

For a simple use case i have just taken one scenario using jdbc in and 
jdbc out, Since the jdbc input source returns the datastream of Row and 
to write them into jdbc database we have to create a table which 
requires schema.

Thanks
Punit


On 02/08/2017 08:22 AM, Chesnay Schepler wrote:
> Hello,
>
> I don't understand why you explicitly need the schema since the batch 
> JDBCInput-/Outputformats don't require it.
> That's kind of the nice thing about Rows.
>
> Would be cool if you could tell us what you're planning to do with the 
> schema :)
>
> In any case, to get the schema within the plan then you will have to 
> query the DB and build it yourself. Note that this
> is executed on the client.
>
> Regards,
> Chesnay
>
> On 08.02.2017 00:39, Punit Tandel wrote:
>>
>> Hi Robert
>>
>> Thanks for the response, So in near future release of the flink 
>> version , is this functionality going to be implemented ?
>>
>> Thanks
>>
>> On 02/07/2017 04:12 PM, Robert Metzger wrote:
>>> Currently, there is no streaming JDBC connector.
>>> Check out this thread from last year: 
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/JDBC-Streaming-Connector-td10508.html
>>>
>>>
>>>
>>> On Mon, Feb 6, 2017 at 5:00 PM, Ufuk Celebi <uce@apache.org 
>>> <mailto:uce@apache.org>> wrote:
>>>
>>>     I'm not sure how well this works for the streaming API. Looping in
>>>     Chesnay, who worked on this.
>>>
>>>     On Mon, Feb 6, 2017 at 11:09 AM, Punit Tandel
>>>     <punit.tandel@ericsson.com> wrote:
>>>     > Hi ,
>>>     >
>>>     > I was looking into flink streaming api and trying to implement
>>>     the solution
>>>     > for reading the data from jdbc database and writing them to
>>>     jdbc databse
>>>     > again.
>>>     >
>>>     > At the moment i can see the datastream is returning Row from
>>>     the database.
>>>     > dataStream.getType().getGenericParameters() retuning an empty
>>>     list of
>>>     > collection.
>>>     >
>>>     > I am right now manually creating a database connection and
>>>     getting the
>>>     > schema from ResultMetadata and constructing the schema for the
>>>     table which
>>>     > is a bit heavy operation.
>>>     >
>>>     > So is there any other way to get the schema for the table in
>>>     order to create
>>>     > a new table and write those records in the database ?
>>>     >
>>>     > Please let me know
>>>     >
>>>     > Thanks
>>>     > Punit
>>>
>>>
>>
>


--------------0C3ACB767B66BDD188E5E11C
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

<html>
  <head>
    <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content-Ty=
pe">
  </head>
  <body bgcolor=3D"#FFFFFF" text=3D"#000000">
    <p>Hi Chesnay</p>
    <p>Currently that is what i have done, reading the schema from
      database in order to create a new table in jdbc database and
      writing the rows coming from jdbcinputformat.</p>
    <p>Overall i am trying to implement the solution which reads the
      streaming data from one source which either could be coming from
      kafka, Jdbc, Hive, Hdfs and writing those streaming data to output
      source which is again could be any of those.</p>
    <p>For a simple use case i have just taken one scenario using jdbc
      in and jdbc out, Since the jdbc input source returns the
      datastream of Row and to write them into jdbc database we have to
      create a table which requires schema.</p>
    <p>Thanks<br>
      Punit<br>
    </p>
    <p><br>
    </p>
    <br>
    <div class=3D"moz-cite-prefix">On 02/08/2017 08:22 AM, Chesnay
      Schepler wrote:<br>
    </div>
    <blockquote cite=3D"mid:589AD528.7000206@apache.org" type=3D"cite">
      <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Du=
tf-8">
      <div class=3D"moz-cite-prefix">Hello,<br>
        <br>
        I don't understand why you explicitly need the schema since the
        batch JDBCInput-/Outputformats don't require it.<br>
        That's kind of the nice thing about Rows.<br>
        <br>
        Would be cool if you could tell us what you're planning to do
        with the schema :)<br>
        <br>
        In any case, to get the schema within the plan then you will
        have to query the DB and build it yourself. Note that this<br>
        is executed on the client.<br>
        <br>
        Regards,<br>
        Chesnay<br>
        <br>
        On 08.02.2017 00:39, Punit Tandel wrote:<br>
      </div>
      <blockquote
        cite=3D"mid:978b13bd-abb7-3954-710f-a48c11aaff93@ericsson.com"
        type=3D"cite">
        <p>Hi Robert</p>
        <p>Thanks for the response, So in near future release of the
          flink version , is this functionality going to be implemented
          ?</p>
        <p>Thanks<br>
        </p>
        <div class=3D"moz-cite-prefix">On 02/07/2017 04:12 PM, Robert
          Metzger wrote:<br>
        </div>
        <blockquote
cite=3D"mid:CAGr9p8D6z1Rm1+yXto8AMUSzCOFqvF1LQc_G9weGub3gHz52uQ@mail.gmai=
l.com"
          type=3D"cite">
          <div dir=3D"ltr">Currently, there is no streaming JDBC
            connector.
            <div>Check out this thread from last year:=C2=A0<a
                moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext"
href=3D"http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/JD=
BC-Streaming-Connector-td10508.html">http://apache-flink-mailing-list-arc=
hive.1008284.n3.nabble.com/JDBC-Streaming-Connector-td10508.html</a><br>
              <br>
              <br>
            </div>
          </div>
          <div class=3D"gmail_extra"><br>
            <div class=3D"gmail_quote">On Mon, Feb 6, 2017 at 5:00 PM,
              Ufuk Celebi <span dir=3D"ltr">&lt;<a moz-do-not-send=3D"tru=
e"
                  href=3D"mailto:uce@apache.org" target=3D"_blank">uce@ap=
ache.org</a>&gt;</span>
              wrote:<br>
              <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
                .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm
                not sure how well this works for the streaming API.
                Looping in<br>
                Chesnay, who worked on this.<br>
                <div class=3D"HOEnZb">
                  <div class=3D"h5"><br>
                    On Mon, Feb 6, 2017 at 11:09 AM, Punit Tandel &lt;<a
                      moz-do-not-send=3D"true"
                      class=3D"moz-txt-link-abbreviated"
                      href=3D"mailto:punit.tandel@ericsson.com">punit.tan=
del@ericsson.com</a>&gt;
                    wrote:<br>
                    &gt; Hi ,<br>
                    &gt;<br>
                    &gt; I was looking into flink streaming api and
                    trying to implement the solution<br>
                    &gt; for reading the data from jdbc database and
                    writing them to jdbc databse<br>
                    &gt; again.<br>
                    &gt;<br>
                    &gt; At the moment i can see the datastream is
                    returning Row from the database.<br>
                    &gt; dataStream.getType().<wbr>getGenericParameters()=

                    retuning an empty list of<br>
                    &gt; collection.<br>
                    &gt;<br>
                    &gt; I am right now manually creating a database
                    connection and getting the<br>
                    &gt; schema from ResultMetadata and constructing the
                    schema for the table which<br>
                    &gt; is a bit heavy operation.<br>
                    &gt;<br>
                    &gt; So is there any other way to get the schema for
                    the table in order to create<br>
                    &gt; a new table and write those records in the
                    database ?<br>
                    &gt;<br>
                    &gt; Please let me know<br>
                    &gt;<br>
                    &gt; Thanks<br>
                    &gt; Punit<br>
                  </div>
                </div>
              </blockquote>
            </div>
            <br>
          </div>
        </blockquote>
        <br>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>

--------------0C3ACB767B66BDD188E5E11C--