Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D8CCF182A8 for ; Fri, 5 Jun 2015 21:11:14 +0000 (UTC) Received: (qmail 38102 invoked by uid 500); 5 Jun 2015 21:11:14 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 38037 invoked by uid 500); 5 Jun 2015 21:11:14 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 38027 invoked by uid 99); 5 Jun 2015 21:11:14 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jun 2015 21:11:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 43E10C126F for ; Fri, 5 Jun 2015 21:11:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4 X-Spam-Level: **** X-Spam-Status: No, score=4 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H2=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id DO710uJxNob5 for ; Fri, 5 Jun 2015 21:11:07 +0000 (UTC) Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com [209.85.160.172]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 1AC15249FD for ; Fri, 5 Jun 2015 21:11:07 +0000 (UTC) Received: by yken206 with SMTP id n206so29515970yke.2 for ; Fri, 05 Jun 2015 14:10:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=9VwPfLLVraG4QL4rbBksQDclq4/L5+2wfeW4k9mIvas=; b=XrLxPTWPYkKL6h4FlVwNLnXi1tjBHPsjWvwIL5cW7KCPhl2LhDS5BoY/sOyIJ1OQ9R iA+6e0+MArAhNt44h6aLsMrsVwlmxcE83q5rJfEjbsmsx2bGMudJ+b+rgOg6LMQxHaCs aIxKpl/h//ptSDKVhlakB4fV+sTP/vuPlpJ0dP04tPSHsCa4W9kw1rTEguvb5I0tey5K kCXDbazWIoom5eO3K3ocFDG27pnv0hXtKYB1oPvg5xbMnD1rW4G551JZVwDY0u0+SX1x menlGjZ2+KQ7e1xLl5tU0nNJp8UQuk7Ut4VIxha5BK5cKTVWg21x71yab7IEDjL/0AAc pEiQ== X-Gm-Message-State: ALoCoQmBKZ3c5nc5OEYU2Np7OrRPI4uCPXcLJbQI/FhwiMe/ZdxPEEf4BCkEgOFipmB1T4UZ9gUC MIME-Version: 1.0 X-Received: by 10.170.165.130 with SMTP id h124mr5589315ykd.23.1433538615598; Fri, 05 Jun 2015 14:10:15 -0700 (PDT) Received: by 10.13.220.135 with HTTP; Fri, 5 Jun 2015 14:10:15 -0700 (PDT) X-Originating-IP: [37.227.43.146] Received: by 10.13.220.135 with HTTP; Fri, 5 Jun 2015 14:10:15 -0700 (PDT) In-Reply-To: References: Date: Fri, 5 Jun 2015 23:10:15 +0200 Message-ID: Subject: Re: Cannot instantiate Mysql connection From: Flavio Pompermaier To: user Content-Type: multipart/alternative; boundary=001a113a95f4f476e10517cbb7a2 --001a113a95f4f476e10517cbb7a2 Content-Type: text/plain; charset=UTF-8 in the fat jar On 5 Jun 2015 19:28, "Stephan Ewen" wrote: > In which way is the driver in the classpath? > > - fat jar? > - in the nested /out folder in the slim jar? > > > > On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier > wrote: > >> Actually I just need to load it in the main method (job manager) before >> calling any flink operation, I retrieve the records in a mysql table >> because they contain the path of files I'll need to read. Nothing more >> nothing less >> On 5 Jun 2015 19:06, "Robert Metzger" wrote: >> >>> Sure. >>> >>> So the DriverManager has a static variable called "registeredDrivers". >>> When DriverManager.getConnection() is called, the method is looking up >>> if an registered driver for that connection (in this case "mysql") is >>> available. >>> >>> For drivers to be in that list, they have to register themselves using >>> the DriverManager.registerDriver() method. >>> >>> Drivers can register themselves with a static constructor (which is >>> executed when Java loads a class): >>> >>> public class PoolingDriver implements Driver { >>> /** Register myself with the {@link DriverManager}. */ >>> static { >>> try { >>> DriverManager.registerDriver(new PoolingDriver()); >>> } catch(Exception e) { >>> } >>> } >>> >>> To execute that driver registration, you need to do: >>> Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver >>> "); >>> because then Java is loading the class and executing the static >>> constructor which is registering the driver at the connection manager. >>> >>> When executing Flink locally, you are using only one JVM. By calling the >>> MySQL driver manually in the main() method of your flink job, you are >>> registering the MySQL driver at the DriverManager of that JVM. >>> >>> However, when you run Flink in a distributed cluster, at the TaskManager >>> JVMs, the MySQL driver is not loaded at the DriverManager there. >>> Therefore, you have to make sure that Class.forName(" >>> org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is >>> not the correct class for the MySQL driver) >>> has been called. >>> One approach to do that is to call Class.forName() in the open() method >>> of your function. >>> >>> Best, >>> Robert >>> >>> >>> On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier >> > wrote: >>> >>>> HI Robert, >>>> In the main method I connect to a mysql table that acts as a >>>> data-source repository that I use to know which dataset I need to load. All >>>> mysql classes are present in the shaded jar. >>>> Could you explain a little bit more in detail the solution to fix this >>>> problem please? Sorry but I didn't understand it :( >>>> >>>> Thanks, >>>> Flavio >>>> On 5 Jun 2015 18:33, "Robert Metzger" wrote: >>>> >>>>> Hi Stefano, >>>>> >>>>> I doubt that there are conflicting dependencies because Flink does not >>>>> contain MySQL dependencies. >>>>> Are you using Flink's JDBCInputFormat or custom code? >>>>> >>>>> For drivers to register at java.sql's DriverManager, their classes >>>>> need to be loaded first. To load a class, you need to call >>>>> Class.forName(""); >>>>> >>>>> Maybe you are loading the class in the application's main() method >>>>> (thats why it is working from eclipse) but not on the cluster instances >>>>> which are supposed to read the data. >>>>> >>>>> On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli >>>>> wrote: >>>>> >>>>>> Hi Robert, >>>>>> >>>>>> I answer on behalf of Flavio. He told me the driver jar was included. >>>>>> Smells lik class-loading issue due to 'conflicting' dependencies. Is it >>>>>> possible? >>>>>> >>>>>> Saluti, >>>>>> Stefano >>>>>> >>>>>> 2015-06-05 16:24 GMT+02:00 Robert Metzger : >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> is the MySQL driver part of the Jar file that you've build? >>>>>>> >>>>>>> On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier < >>>>>>> pompermaier@okkam.it> wrote: >>>>>>> >>>>>>>> Hi to all, >>>>>>>> >>>>>>>> I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I >>>>>>>> set up a mysql connection. >>>>>>>> When I run the job from Eclipse everything is fine, >>>>>>>> while when running the job from the Web UI I get the following >>>>>>>> exception: >>>>>>>> >>>>>>>> java.sql.SQLException: No suitable driver found for >>>>>>>> jdbc:mysql:/localhost:3306/mydb?autoReconnect=true >>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:596) >>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:215) >>>>>>>> >>>>>>>> How can I fix that? >>>>>>>> >>>>>>>> Best, >>>>>>>> Flavio >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>> > --001a113a95f4f476e10517cbb7a2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

in the fat jar

On 5 Jun 2015 19:28, "Stephan Ewen" &l= t;sewen@apache.org> wrote:
In which= way is the driver in the classpath?

=C2=A0- fat jar?
=C2=A0- in the nested /out folder in the slim jar?



On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <pompermaier= @okkam.it> wrote:

Actually I just need to load it in the main method (job manager) b= efore calling any flink operation, I retrieve the records in a mysql table = because they contain the path of files I'll need to read. Nothing more = nothing less

On 5 Jun 2015 19:06, "Robert Metzger" = <rmetzger@apach= e.org> wrote:
Sure.

So the=C2=A0DriverManager has a static variable called=C2=A0"registeredDrivers".
When=C2=A0DriverManager.getConnection() is called, t= he method is looking up if an registered driver for that connection (in thi= s case "mysql") is available.

For driver= s to be in that list, they have to register themselves using the=C2=A0Drive= rManager.registerDriver() method.

Drivers can regi= ster themselves with a static constructor (which is executed when Java load= s a class):

public class PoolingDriver implements Driver {
    /** Register myself with=
 the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {        }
    }
To execute that driver registration, yo= u need to do:
Class.forName("org.datanucleus.stor= e.rdbms.datasource.dbcp.PoolingDriver");
because then Java is loading the class and executing the static const= ructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. B= y calling the MySQL driver manually in the main() method of your flink job,= you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster,= at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManag= er there.
Therefore, you have to make sure that Class.forName(&qu= ot;org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL = driver)
has been called.
One approach to do that is to = call Class.forName() in the open() method of your function.

<= /div>
Best,
Robert


On Fri, Jun 5, 2015 at 6:54 PM, F= lavio Pompermaier <pompermaier@okkam.it> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source re= pository that I use to know which dataset I need to load. All mysql classes= are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this prob= lem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" = <rmetzger@apach= e.org> wrote:
Hi Stefano,

I doubt that there are con= flicting dependencies because Flink does not contain MySQL dependencies.
Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, th= eir classes need to be loaded first. To load a class, you need to call Clas= s.forName("<classname>");

Maybe you are loading the = class in the application's main() method (thats why it is working from = eclipse) but not on the cluster instances which are supposed to read the da= ta.

On= Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <s.bortoli@gmail.com&g= t; wrote:
Hi Robert,

I answer on behalf of Flavio. He told me th= e driver jar was included. Smells lik class-loading issue due to 'confl= icting' dependencies.=C2=A0 Is it possible?

Saluti,
Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger &l= t;rmetzger@apache.= org>:
Hi,<= div>
is the MySQL driver part of the Jar file that you've= build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <pomperm= aier@okkam.it> wrote:
Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job = I set up a mysql connection.
When I run the job from Eclipse ever= ything is fine,
while when running the job from the Web UI I get = the following exception:

java.sql.SQLException: No suitable driver= found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=3Dtrue
at java= .sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.Dr= iverManager.getConnection(DriverManager.java:215)

How ca= n I fix that?

Best,
Flavio





--001a113a95f4f476e10517cbb7a2--