flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Re: Checking actual config values used by TaskManager
Date Fri, 29 Apr 2016 13:18:31 GMT
Hi Timur,

> On Apr 28, 2016, at 10:40pm, Timur Fayruzov <timur.fairuzov@gmail.com> wrote:
> 
> If you're talking about parameters that were set on JVM startup then `ps aux|grep flink`
on an EMR slave node should do the trick, that'll give you the full command line.

No, I’m talking about values that come from flink-conf.yaml.

Maybe there’s no good reason to worry, but in Hadoop land you can have parameters set via
the conf on the client, which in turn get overridden by values from conf files on the nodes,
which you can then override via command line parameters, which in turn can be changed by the
user code.

Plus parameters that can be flagged as final/unmodifiable, and thus some of the above actually
don’t change anything.

So it’s a common issue where what you think you set as a value isn’t actually being used,
and that’s why examining the job conf that was actually deployed with tasks is critical.

— Ken


> 
> On Thu, Apr 28, 2016 at 9:00 PM, Ken Krugler <kkrugler_lists@transpac.com <mailto:kkrugler_lists@transpac.com>>
wrote:
> Hi all,
> 
> I’m running jobs on EMR via YARN, and wondering how to check exactly what configuration
settings are actually being used.
> 
> This is mostly for the TaskManager.
> 
> I know I can modify the conf/flink-conf.yaml file, and (via the CLI) I can use -yD param=value.
> 
> But my experience with Hadoop makes me want to see the exact values being used, versus
assuming I know what’s been set :)
> 
> Thanks,
> 
> — Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr




Mime
View raw message