Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71109182BB for ; Fri, 5 Jun 2015 21:13:29 +0000 (UTC) Received: (qmail 41367 invoked by uid 500); 5 Jun 2015 21:13:29 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 41301 invoked by uid 500); 5 Jun 2015 21:13:29 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 41291 invoked by uid 99); 5 Jun 2015 21:13:29 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jun 2015 21:13:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B28EBC126F for ; Fri, 5 Jun 2015 21:13:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.001 X-Spam-Level: *** X-Spam-Status: No, score=3.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id UA9AU65b9p4g for ; Fri, 5 Jun 2015 21:13:16 +0000 (UTC) Received: from mail-ig0-f182.google.com (mail-ig0-f182.google.com [209.85.213.182]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id A8B4D275E7 for ; Fri, 5 Jun 2015 21:13:15 +0000 (UTC) Received: by igbzc4 with SMTP id zc4so25312018igb.0 for ; Fri, 05 Jun 2015 14:13:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=XgFzasSffqREgMyuCJxvusZu6v5vNNGZ4iK8GqQx5Kk=; b=tYmY81YnTCTmN2YpJmErRwIBV/AumKpm6gV3InjNlJIVkmbcf4d5WLCnI/R68TnoA+ 6U/obb/hsxWU9sbEeE2klfzGmSO+rGsnm99D97NYRwbonVR0MHBTHFHAHhQV93szOgIH 0hRDjpkafD/f7qHDgJq4GhZwUkgzK+MJa9Uk9rkrkysQWYNm2h8+UZG3oFEAyHsHSZsS YVanKYVakuNnnJE+jjppHCyy66kconcC1coZ5GdWII0d4aSnaxRAYfZTNUEh5xJEO6Ab 4tLubZ+wllvmpNyeaYI9rzxMi0wQ9ShndPB/6eoXcxQIuIcFbMES0Jed6blajUJt+Sks lU4A== MIME-Version: 1.0 X-Received: by 10.43.140.5 with SMTP id iy5mr12901038icc.77.1433538795039; Fri, 05 Jun 2015 14:13:15 -0700 (PDT) Sender: ewenstephan@gmail.com Received: by 10.64.241.230 with HTTP; Fri, 5 Jun 2015 14:13:14 -0700 (PDT) In-Reply-To: References: Date: Fri, 5 Jun 2015 23:13:14 +0200 X-Google-Sender-Auth: _PyRVvlhfxmAGaUIAF4g-p49R1I Message-ID: Subject: Re: Cannot instantiate Mysql connection From: Stephan Ewen To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a11c2c588a675ac0517cbc2a3 --001a11c2c588a675ac0517cbc2a3 Content-Type: text/plain; charset=UTF-8 Can you manually load the driver class, with "Class.forName(...)", or does that yield a "ClassNotFoundException" ? On Fri, Jun 5, 2015 at 11:10 PM, Flavio Pompermaier wrote: > in the fat jar > On 5 Jun 2015 19:28, "Stephan Ewen" wrote: > >> In which way is the driver in the classpath? >> >> - fat jar? >> - in the nested /out folder in the slim jar? >> >> >> >> On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier >> wrote: >> >>> Actually I just need to load it in the main method (job manager) before >>> calling any flink operation, I retrieve the records in a mysql table >>> because they contain the path of files I'll need to read. Nothing more >>> nothing less >>> On 5 Jun 2015 19:06, "Robert Metzger" wrote: >>> >>>> Sure. >>>> >>>> So the DriverManager has a static variable called "registeredDrivers". >>>> When DriverManager.getConnection() is called, the method is looking up >>>> if an registered driver for that connection (in this case "mysql") is >>>> available. >>>> >>>> For drivers to be in that list, they have to register themselves using >>>> the DriverManager.registerDriver() method. >>>> >>>> Drivers can register themselves with a static constructor (which is >>>> executed when Java loads a class): >>>> >>>> public class PoolingDriver implements Driver { >>>> /** Register myself with the {@link DriverManager}. */ >>>> static { >>>> try { >>>> DriverManager.registerDriver(new PoolingDriver()); >>>> } catch(Exception e) { >>>> } >>>> } >>>> >>>> To execute that driver registration, you need to do: >>>> Class.forName("org.datanucleus.store.rdbms.datasource.dbcp. >>>> PoolingDriver"); >>>> because then Java is loading the class and executing the static >>>> constructor which is registering the driver at the connection manager. >>>> >>>> When executing Flink locally, you are using only one JVM. By calling >>>> the MySQL driver manually in the main() method of your flink job, you are >>>> registering the MySQL driver at the DriverManager of that JVM. >>>> >>>> However, when you run Flink in a distributed cluster, at the >>>> TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there. >>>> Therefore, you have to make sure that Class.forName(" >>>> org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is >>>> not the correct class for the MySQL driver) >>>> has been called. >>>> One approach to do that is to call Class.forName() in the open() method >>>> of your function. >>>> >>>> Best, >>>> Robert >>>> >>>> >>>> On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier < >>>> pompermaier@okkam.it> wrote: >>>> >>>>> HI Robert, >>>>> In the main method I connect to a mysql table that acts as a >>>>> data-source repository that I use to know which dataset I need to load. All >>>>> mysql classes are present in the shaded jar. >>>>> Could you explain a little bit more in detail the solution to fix this >>>>> problem please? Sorry but I didn't understand it :( >>>>> >>>>> Thanks, >>>>> Flavio >>>>> On 5 Jun 2015 18:33, "Robert Metzger" wrote: >>>>> >>>>>> Hi Stefano, >>>>>> >>>>>> I doubt that there are conflicting dependencies because Flink does >>>>>> not contain MySQL dependencies. >>>>>> Are you using Flink's JDBCInputFormat or custom code? >>>>>> >>>>>> For drivers to register at java.sql's DriverManager, their classes >>>>>> need to be loaded first. To load a class, you need to call >>>>>> Class.forName(""); >>>>>> >>>>>> Maybe you are loading the class in the application's main() method >>>>>> (thats why it is working from eclipse) but not on the cluster instances >>>>>> which are supposed to read the data. >>>>>> >>>>>> On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli >>>>>> wrote: >>>>>> >>>>>>> Hi Robert, >>>>>>> >>>>>>> I answer on behalf of Flavio. He told me the driver jar was >>>>>>> included. Smells lik class-loading issue due to 'conflicting' >>>>>>> dependencies. Is it possible? >>>>>>> >>>>>>> Saluti, >>>>>>> Stefano >>>>>>> >>>>>>> 2015-06-05 16:24 GMT+02:00 Robert Metzger : >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> is the MySQL driver part of the Jar file that you've build? >>>>>>>> >>>>>>>> On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier < >>>>>>>> pompermaier@okkam.it> wrote: >>>>>>>> >>>>>>>>> Hi to all, >>>>>>>>> >>>>>>>>> I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job >>>>>>>>> I set up a mysql connection. >>>>>>>>> When I run the job from Eclipse everything is fine, >>>>>>>>> while when running the job from the Web UI I get the following >>>>>>>>> exception: >>>>>>>>> >>>>>>>>> java.sql.SQLException: No suitable driver found for >>>>>>>>> jdbc:mysql:/localhost:3306/mydb?autoReconnect=true >>>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:596) >>>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:215) >>>>>>>>> >>>>>>>>> How can I fix that? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Flavio >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >> --001a11c2c588a675ac0517cbc2a3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Can you manually load the driver class, with "Class.f= orName(...)", or does that yield a "ClassNotFoundException" = ?

On Fri, Ju= n 5, 2015 at 11:10 PM, Flavio Pompermaier <pompermaier@okkam.it>= wrote:

in the fat = jar

On 5 Jun 2015 19:28, "Stephan Ewen" &l= t;sewen@apache.org> wrote:
In which way is the driver in the classpath?

= =C2=A0- fat jar?
=C2=A0- in the nested /out folder in the slim ja= r?



=
On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermai= er <pompermaier@okkam.it> wrote:

Actually I just need to load it in the main method= (job manager) before calling any flink operation, I retrieve the records i= n a mysql table because they contain the path of files I'll need to rea= d. Nothing more nothing less

On 5 Jun 2015 19:06, "Robert Metzger" = <rmetzger@apach= e.org> wrote:
Sure.

So the=C2=A0DriverManager has a static variable called=C2=A0"registeredDrivers".
When=C2=A0DriverManager.getConnection() is called, t= he method is looking up if an registered driver for that connection (in thi= s case "mysql") is available.

For driver= s to be in that list, they have to register themselves using the=C2=A0Drive= rManager.registerDriver() method.

Drivers can regi= ster themselves with a static constructor (which is executed when Java load= s a class):

public class PoolingDriver implements Driver {
    /** Register myself with=
 the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {        }
    }
To execute that driver registration, yo= u need to do:
Class.forName("org.datanucleus.stor= e.rdbms.datasource.dbcp.PoolingDriver");
because then Java is loading the class and executing the static const= ructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. B= y calling the MySQL driver manually in the main() method of your flink job,= you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster,= at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManag= er there.
Therefore, you have to make sure that Class.forName(&qu= ot;org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL = driver)
has been called.
One approach to do that is to = call Class.forName() in the open() method of your function.

<= /div>
Best,
Robert


On Fri, Jun 5, 2015 at 6:54 PM, F= lavio Pompermaier <pompermaier@okkam.it> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source re= pository that I use to know which dataset I need to load. All mysql classes= are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this prob= lem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" = <rmetzger@apach= e.org> wrote:
Hi Stefano,

I doubt that there are con= flicting dependencies because Flink does not contain MySQL dependencies.
Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, th= eir classes need to be loaded first. To load a class, you need to call Clas= s.forName("<classname>");

Maybe you are loading the = class in the application's main() method (thats why it is working from = eclipse) but not on the cluster instances which are supposed to read the da= ta.

On= Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <s.bortoli@gmail.com&g= t; wrote:
Hi Robert,

I answer on behalf of Flavio. He told me th= e driver jar was included. Smells lik class-loading issue due to 'confl= icting' dependencies.=C2=A0 Is it possible?

Saluti,
Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger &l= t;rmetzger@apache.= org>:
Hi,<= div>
is the MySQL driver part of the Jar file that you've= build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <pomperm= aier@okkam.it> wrote:
Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job = I set up a mysql connection.
When I run the job from Eclipse ever= ything is fine,
while when running the job from the Web UI I get = the following exception:

java.sql.SQLException: No suitable driver= found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=3Dtrue
at java= .sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.Dr= iverManager.getConnection(DriverManager.java:215)

How ca= n I fix that?

Best,
Flavio






--001a11c2c588a675ac0517cbc2a3--