flink-user-zh mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhenghua Gao <doc...@gmail.com>
Subject Re: JDBCLookupFunction被缓存导致数据的不及时性
Date Wed, 15 Apr 2020 03:38:56 GMT
有两个参数可以控制cache大小和cache失效时间 [1],你可以在性能和准确性上做权衡

  -- lookup options, optional, used in temporary join
  'connector.lookup.cache.max-rows' = '5000', -- optional, max number
of rows of lookup cache, over this value, the oldest rows will
                                              -- be eliminated.
"cache.max-rows" and "cache.ttl" options must all be specified if any
                                              -- of them is specified.
Cache is not enabled as default.
  'connector.lookup.cache.ttl' = '10s', -- optional, the max time to
live for each rows in lookup cache, over this time, the oldest rows
                                        -- will be expired.
"cache.max-rows" and "cache.ttl" options must all be specified if any
of
                                        -- them is specified. Cache is
not enabled as default.
  'connector.lookup.max-retries' = '3', -- optional, max retry times
if lookup database failed

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#jdbc-connector

*Best Regards,*
*Zhenghua Gao*


On Wed, Apr 15, 2020 at 11:28 AM Dino Zhang <victordinozhang@gmail.com>
wrote:

> 可以考虑调小cache.ttl
>
> On Wed, Apr 15, 2020 at 11:22 AM tingli ke <ketingli92@gmail.com> wrote:
>
> > 是否有其他的方式来对mysql维表数据进行实时join
> >
> >
> > 13122260573@163.com <13122260573@163.com> 于2020年4月15日周三 上午11:08写道:
> >
> > > 有个setCacheMaxSize(1000),可以改成 -1 表示不使用cache
> > > org.apache.flink.api.java.io.jdbc.JDBCLookupFunction 这个方法有解释
> > > The cacheMaxSize is -1 means not use cache
> > >
> > >
> > >
> > > 13122260573@163.com
> > >
> > > 发件人: tingli ke
> > > 发送时间: 2020-04-15 10:55
> > > 收件人: user-zh
> > > 主题: JDBCLookupFunction被缓存导致数据的不及时性
> > > Hi,
> > >
> > >
> >
> 流表通过JDBCLookupFunction来对mysql的维表数据进行实时join,但是JDBCLookupFunction会对数据进行缓存,导致mysql的维表数据被更新,但是flink还是老的数据,(考虑性能需要被缓存)
> > > 是否有其他的方式来对mysql维表数据进行实时join
> > >
> >
>
>
> --
> Regards,
> DinoZhang
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message