flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Cannot instantiate Mysql connection
Date Fri, 05 Jun 2015 21:13:14 GMT
Can you manually load the driver class, with "Class.forName(...)", or does
that yield a "ClassNotFoundException" ?

On Fri, Jun 5, 2015 at 11:10 PM, Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> in the fat jar
> On 5 Jun 2015 19:28, "Stephan Ewen" <sewen@apache.org> wrote:
>
>> In which way is the driver in the classpath?
>>
>>  - fat jar?
>>  - in the nested /out folder in the slim jar?
>>
>>
>>
>> On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <pompermaier@okkam.it>
>> wrote:
>>
>>> Actually I just need to load it in the main method (job manager) before
>>> calling any flink operation, I retrieve the records in a mysql table
>>> because they contain the path of files I'll need to read. Nothing more
>>> nothing less
>>> On 5 Jun 2015 19:06, "Robert Metzger" <rmetzger@apache.org> wrote:
>>>
>>>> Sure.
>>>>
>>>> So the DriverManager has a static variable called "registeredDrivers".
>>>> When DriverManager.getConnection() is called, the method is looking up
>>>> if an registered driver for that connection (in this case "mysql") is
>>>> available.
>>>>
>>>> For drivers to be in that list, they have to register themselves using
>>>> the DriverManager.registerDriver() method.
>>>>
>>>> Drivers can register themselves with a static constructor (which is
>>>> executed when Java loads a class):
>>>>
>>>> public class PoolingDriver implements Driver {
>>>>     /** Register myself with the {@link DriverManager}. */
>>>>     static {
>>>>         try {
>>>>             DriverManager.registerDriver(new PoolingDriver());
>>>>         } catch(Exception e) {
>>>>         }
>>>>     }
>>>>
>>>> To execute that driver registration, you need to do:
>>>> Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.
>>>> PoolingDriver");
>>>> because then Java is loading the class and executing the static
>>>> constructor which is registering the driver at the connection manager.
>>>>
>>>> When executing Flink locally, you are using only one JVM. By calling
>>>> the MySQL driver manually in the main() method of your flink job, you are
>>>> registering the MySQL driver at the DriverManager of that JVM.
>>>>
>>>> However, when you run Flink in a distributed cluster, at the
>>>> TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.
>>>> Therefore, you have to make sure that Class.forName("
>>>> org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is
>>>> not the correct class for the MySQL driver)
>>>> has been called.
>>>> One approach to do that is to call Class.forName() in the open() method
>>>> of your function.
>>>>
>>>> Best,
>>>> Robert
>>>>
>>>>
>>>> On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> HI Robert,
>>>>> In the main method I connect to a mysql table that acts as a
>>>>> data-source repository that I use to know which dataset I need to load.
All
>>>>> mysql classes are present in the shaded jar.
>>>>> Could you explain a little bit more in detail the solution to fix this
>>>>> problem please? Sorry but I didn't understand it :(
>>>>>
>>>>> Thanks,
>>>>> Flavio
>>>>> On 5 Jun 2015 18:33, "Robert Metzger" <rmetzger@apache.org> wrote:
>>>>>
>>>>>> Hi Stefano,
>>>>>>
>>>>>> I doubt that there are conflicting dependencies because Flink does
>>>>>> not contain MySQL dependencies.
>>>>>> Are you using Flink's JDBCInputFormat or custom code?
>>>>>>
>>>>>> For drivers to register at java.sql's DriverManager, their classes
>>>>>> need to be loaded first. To load a class, you need to call
>>>>>> Class.forName("<classname>");
>>>>>>
>>>>>> Maybe you are loading the class in the application's main() method
>>>>>> (thats why it is working from eclipse) but not on the cluster instances
>>>>>> which are supposed to read the data.
>>>>>>
>>>>>> On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <s.bortoli@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Robert,
>>>>>>>
>>>>>>> I answer on behalf of Flavio. He told me the driver jar was
>>>>>>> included. Smells lik class-loading issue due to 'conflicting'
>>>>>>> dependencies.  Is it possible?
>>>>>>>
>>>>>>> Saluti,
>>>>>>> Stefano
>>>>>>>
>>>>>>> 2015-06-05 16:24 GMT+02:00 Robert Metzger <rmetzger@apache.org>:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> is the MySQL driver part of the Jar file that you've build?
>>>>>>>>
>>>>>>>> On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <
>>>>>>>> pompermaier@okkam.it> wrote:
>>>>>>>>
>>>>>>>>> Hi to all,
>>>>>>>>>
>>>>>>>>> I'm using a fresh build of flink-0.9-SNAPSHOT and in
my flink job
>>>>>>>>> I set up a mysql connection.
>>>>>>>>> When I run the job from Eclipse everything is fine,
>>>>>>>>> while when running the job from the Web UI I get the
following
>>>>>>>>> exception:
>>>>>>>>>
>>>>>>>>> java.sql.SQLException: No suitable driver found for
>>>>>>>>> jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
>>>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:596)
>>>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:215)
>>>>>>>>>
>>>>>>>>> How can I fix that?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Flavio
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>

Mime
View raw message