flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mauro Cortellazzi <mauro.cortella...@radicalbit.io>
Subject Re: Question about jobmanager.web.upload.dir
Date Thu, 18 May 2017 06:32:29 GMT
Thank you Chesnay and Timo for your answers.

Il 17/05/2017 16:49, Chesnay Schepler ha scritto:
> I don't know why we delete it either, but my guess is that at one 
> point this was a temporary directory that we properly cleaned up, and 
> later allowed it to be configurable.
> Currently, this directory must be in the local file system. We could 
> change it to also allow non-local paths, which should be fairly easy 
> to do. Add a condition that the directory is only cleaned up when it 
> wasn't explicitly configured and we should've covered everything.
> In regards to the HistoryServer, i mean of course could we include the 
> user-jar, but I'm wondering about the user-cases for it.
> It's not like you could "just download it and give it to a 
> JobManager"; you would be missing any additional libraries put under 
> /lib, as well as the original configuration of the cluster. It would 
> also require extra steps (and tools) to identify which dependencies 
> are required.
> And there are downsides to this, like a significant increase in the 
> size of the job archives (along with a duplication of jars for every 
> job submission), and the archiving also becomes more complex.
> On 17.05.2017 15:52, Timo Walther wrote:
>> Hey Mauro,
>> I'm not aware of any reason for that. I loop in Chesnay, maybe he 
>> knows why. @Chesnay wouldn't it be helpful to also archive the jars 
>> using the HistoryServer?
>> Timo
>> Am 17.05.17 um 12:31 schrieb Mauro Cortellazzi:
>>> Hi Flink comunity,
>>> is there a particular reason to delete the jobmanager.web.upload.dir 
>>> content when the JobManager restart? Could it be interesting have 
>>> the jar previously uploaded when JM restart? Could the jars be saved 
>>> to HDFS for HA mode?
>>> Thank you for your support.
>>> Mauro

View raw message