flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From miki haiat <miko5...@gmail.com>
Subject Re: Query big mssql Data Source [Batch]
Date Thu, 06 Dec 2018 14:49:05 GMT
Hi Flavio ,
That working fine for and im able to pull ~17m rows in 20 seconds.

Im a bit confuse regarding the state backhand ,
I could find a way to configure it so im guessing the data is in the memory
...

thanks,
Miki



On Thu, Dec 6, 2018 at 12:06 PM Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> the constructor of NumericBetweenParametersProvider takes 3 params: long
> fetchSize, long minVal, long maxVal.
> If you want parallelism you should use a  1 < fetchSize  < maxVal.
> In your case, if you do new NumericBetweenParametersProvider(50, 3, 300)
> you will produce 6 parallel tasks:
>
>    1. SELECT .... BETWEEN 3 and 50
>    2. SELECT .... BETWEEN 51 and 100
>    3. SELECT .... BETWEEN 101 and 150
>    4. SELECT .... BETWEEN 151 and 200
>    5. SELECT .... BETWEEN 201 and 250
>    6. SELECT .... BETWEEN 251 and 300
>
>
> On Thu, Dec 6, 2018 at 10:32 AM miki haiat <miko5054@gmail.com> wrote:
>
>> hi Flavio ,
>>
>> This is the query that im trying to coordinate
>>
>>> .setQuery("SELECT a, b, c, \n" +
>>>         "FROM dbx.dbo.x as tls\n"+
>>>         "WHERE tls.a BETWEEN ? and ?"
>>>
>>> And this is the way im trying to parameterized
>>
>> ParameterValuesProvider pramProvider = new
>> NumericBetweenParametersProvider(10000, 3,300);
>>
>> I also tried this way
>>
>>  Serializable[][] queryParameters = new String[1][2];
>> queryParameters[0] = new String[]{"3","300"};
>>
>>
>> On Wed, Dec 5, 2018 at 6:44 PM Flavio Pompermaier <pompermaier@okkam.it>
>> wrote:
>>
>>> whats your query? Have you used '?' where query should be parameterized?
>>>
>>> Give a look at
>>> https://github.com/apache/flink/blob/master/flink-connectors/flink-jdbc/src/test/java/org/apache/flink/api/java/io/jdbc/JDBCFullTest.java
>>>
>>
>

Mime
View raw message