flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Richter <s.rich...@data-artisans.com>
Subject Re: Support for multiple HDFS
Date Thu, 24 Aug 2017 09:47:19 GMT
Hi,

I don’t think that this is currently supported. If you see a use case for this (over creating
different root directories for checkpoint data and result data) then I suggest that you open
a JIRA issue with a new feature request.

Best,
Stefan

> Am 23.08.2017 um 20:17 schrieb Vijay Srinivasaraghavan <vijikarthi@yahoo.com>:
> 
> Hi Ted,
> 
> I believe HDFS-6584 is more of an HDFS feature supporting archive use case through some
policy configurations.
> 
> My ask is that I have two distinct HCFS File systems which are independent but the Flink
job will decide which one to use for sink while the Flink infrastructure is by default configured
with one of these HCFS as state backend store.
> 
> Hope this helps.
> 
> Regards
> Vijay
> 
> 
> On Wednesday, August 23, 2017 11:06 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> 
> 
> Would HDFS-6584 help with your use case ?
> 
> On Wed, Aug 23, 2017 at 11:00 AM, Vijay Srinivasaraghavan <
> vijikarthi@yahoo.com.invalid <mailto:vijikarthi@yahoo.com.invalid>> wrote:
> 
> > Hello,
> > Is it possible for a Flink cluster to use multiple HDFS repository (HDFS-1
> > for managing Flink state backend, HDFS-2 for syncing results from user
> > job)?
> > The scenario can be viewed in the context of running some jobs that are
> > meant to push the results to an archive repository (cold storage).
> > Since the hadoop configuration is static, I am thinking it is hard to
> > achieve this but I could be wrong.
> > Please share any thoughts.
> > RegardsVijay
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message