flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bowen Li (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-7365) excessive warning logs of attempt to override final parameter: fs.s3.buffer.dir
Date Fri, 04 Aug 2017 17:00:05 GMT

     [ https://issues.apache.org/jira/browse/FLINK-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Bowen Li updated FLINK-7365:
----------------------------
    Description: 
I'm seeing hundreds of line of the following log in my JobManager log file:


{code:java}
2017-08-03 19:48:45,330 WARN  org.apache.hadoop.conf.Configuration                       
  - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring.
2017-08-03 19:48:45,485 WARN  org.apache.hadoop.conf.Configuration                       
  - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring.
2017-08-03 19:48:45,486 WARN  org.apache.hadoop.conf.Configuration                       
  - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring.
2017-08-03 19:48:45,626 WARN  org.apache.hadoop.conf.Configuration                       
  - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring
......
{code}

Info of my Flink cluster:

- Running on EMR with emr-5.6.0
- Using FSStateBackend, writing checkpointing data files to s3
- Configured s3 with S3AFileSystem according to https://ci.apache.org/projects/flink/flink-docs-release-1.4/setup/aws.html#set-s3-filesystem
- AWS forbids resetting 'fs.s3.buffer.dir' value (it has a <final> tag on this property
in core-site.xml), so I set 'fs.s3a.buffer.dir' as '/tmp'

Here's my core-site.xml file:


{code:java}
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Licensed to the Apache Software Foundation (ASF) under one or more       -->
<!-- contributor license agreements.  See the NOTICE file distributed with    -->
<!-- this work for additional information regarding copyright ownership.      -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0  -->
<!-- (the "License"); you may not use this file except in compliance with     -->
<!-- the License.  You may obtain a copy of the License at                    -->
<!--                                                                          -->
<!--     http://www.apache.org/licenses/LICENSE-2.0                           -->
<!--                                                                          -->
<!-- Unless required by applicable law or agreed to in writing, software      -->
<!-- distributed under the License is distributed on an "AS IS" BASIS,        -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and      -->
<!-- limitations under the License.                                           -->

<configuration>
<property>
  <name>fs.s3.buffer.dir</name>
  <value>/mnt/s3,/mnt1/s3</value>
  <final>true</final>
</property>

<property>
  <name>fs.s3.impl</name>
  <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>

<property>
  <name>fs.s3n.impl</name>
  <value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value>
</property>

  <property>
    <name>ipc.client.connect.max.retries.on.timeouts</name>
    <value>5</value>
  </property>

  <property>
    <name>hadoop.security.key.default.bitlength</name>
    <value>256</value>
  </property>

  <property>
    <name>hadoop.proxyuser.hadoop.groups</name>
    <value>*</value>
  </property>

  <property>
    <name>hadoop.tmp.dir</name>
    <value>/mnt/var/lib/hadoop/tmp</value>
  </property>

  <property>
    <name>hadoop.proxyuser.hadoop.hosts</name>
    <value>*</value>
  </property>

  <property>
    <name>io.file.buffer.size</name>
    <value>65536</value>
  </property>

  <property>
    <name>fs.AbstractFileSystem.s3.impl</name>
    <value>org.apache.hadoop.fs.s3.EMRFSDelegate</value>
  </property>

  <property>
    <name>fs.s3a.buffer.dir</name>
    <value>/tmp</value>
  </property>

  <property>
    <name>fs.s3bfs.impl</name>
    <value>org.apache.hadoop.fs.s3.S3FileSystem</value>
  </property>

</configuration>
{code}

This bug is about excessive logging.


  was:
I'm seeing hundreds of line of the following log in my JobManager log file:


{code:java}
2017-08-03 19:48:45,330 WARN  org.apache.hadoop.conf.Configuration                       
  - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring.
2017-08-03 19:48:45,485 WARN  org.apache.hadoop.conf.Configuration                       
  - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring.
2017-08-03 19:48:45,486 WARN  org.apache.hadoop.conf.Configuration                       
  - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring.
2017-08-03 19:48:45,626 WARN  org.apache.hadoop.conf.Configuration                       
  - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring
......
{code}

Info of my Flink cluster:

- Running on EMR with emr-5.6.0
- Using FSStateBackend, writing checkpointing data files to s3
- Configured s3 with S3AFileSystem according to https://ci.apache.org/projects/flink/flink-docs-release-1.4/setup/aws.html#set-s3-filesystem
- AWS forbids resetting 'fs.s3.buffer.dir' value (it has a <final> tag on this property
in core-site.xml), so I set 'fs.s3a.buffer.dir' as '/tmp'

Here's my core-site.xml file:


{code:java}
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Licensed to the Apache Software Foundation (ASF) under one or more       -->
<!-- contributor license agreements.  See the NOTICE file distributed with    -->
<!-- this work for additional information regarding copyright ownership.      -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0  -->
<!-- (the "License"); you may not use this file except in compliance with     -->
<!-- the License.  You may obtain a copy of the License at                    -->
<!--                                                                          -->
<!--     http://www.apache.org/licenses/LICENSE-2.0                           -->
<!--                                                                          -->
<!-- Unless required by applicable law or agreed to in writing, software      -->
<!-- distributed under the License is distributed on an "AS IS" BASIS,        -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and      -->
<!-- limitations under the License.                                           -->

<configuration>
<property>
  <name>fs.s3.buffer.dir</name>
  <value>/mnt/s3,/mnt1/s3</value>
  <final>true</final>
</property>

<property>
  <name>fs.s3.impl</name>
  <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>

<property>
  <name>fs.s3n.impl</name>
  <value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value>
</property>

  <property>
    <name>ipc.client.connect.max.retries.on.timeouts</name>
    <value>5</value>
  </property>

  <property>
    <name>hadoop.security.key.default.bitlength</name>
    <value>256</value>
  </property>

  <property>
    <name>hadoop.proxyuser.hadoop.groups</name>
    <value>*</value>
  </property>

  <property>
    <name>hadoop.tmp.dir</name>
    <value>/mnt/var/lib/hadoop/tmp</value>
  </property>

  <property>
    <name>hadoop.proxyuser.hadoop.hosts</name>
    <value>*</value>
  </property>

  <property>
    <name>io.file.buffer.size</name>
    <value>65536</value>
  </property>

  <property>
    <name>fs.AbstractFileSystem.s3.impl</name>
    <value>org.apache.hadoop.fs.s3.EMRFSDelegate</value>
  </property>

  <property>
    <name>fs.s3a.buffer.dir</name>
    <value>/tmp</value>
  </property>

  <property>
    <name>fs.s3bfs.impl</name>
    <value>org.apache.hadoop.fs.s3.S3FileSystem</value>
  </property>

</configuration>
{code}





> excessive warning logs of attempt to override final parameter: fs.s3.buffer.dir
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-7365
>                 URL: https://issues.apache.org/jira/browse/FLINK-7365
>             Project: Flink
>          Issue Type: Bug
>          Components: Configuration
>    Affects Versions: 1.3.0
>            Reporter: Bowen Li
>
> I'm seeing hundreds of line of the following log in my JobManager log file:
> {code:java}
> 2017-08-03 19:48:45,330 WARN  org.apache.hadoop.conf.Configuration                  
       - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter:
fs.s3.buffer.dir;  Ignoring.
> 2017-08-03 19:48:45,485 WARN  org.apache.hadoop.conf.Configuration                  
       - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring.
> 2017-08-03 19:48:45,486 WARN  org.apache.hadoop.conf.Configuration                  
       - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter:
fs.s3.buffer.dir;  Ignoring.
> 2017-08-03 19:48:45,626 WARN  org.apache.hadoop.conf.Configuration                  
       - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir;
 Ignoring
> ......
> {code}
> Info of my Flink cluster:
> - Running on EMR with emr-5.6.0
> - Using FSStateBackend, writing checkpointing data files to s3
> - Configured s3 with S3AFileSystem according to https://ci.apache.org/projects/flink/flink-docs-release-1.4/setup/aws.html#set-s3-filesystem
> - AWS forbids resetting 'fs.s3.buffer.dir' value (it has a <final> tag on this
property in core-site.xml), so I set 'fs.s3a.buffer.dir' as '/tmp'
> Here's my core-site.xml file:
> {code:java}
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <!-- Licensed to the Apache Software Foundation (ASF) under one or more       -->
> <!-- contributor license agreements.  See the NOTICE file distributed with    -->
> <!-- this work for additional information regarding copyright ownership.      -->
> <!-- The ASF licenses this file to You under the Apache License, Version 2.0  -->
> <!-- (the "License"); you may not use this file except in compliance with     -->
> <!-- the License.  You may obtain a copy of the License at                    -->
> <!--                                                                          -->
> <!--     http://www.apache.org/licenses/LICENSE-2.0                           -->
> <!--                                                                          -->
> <!-- Unless required by applicable law or agreed to in writing, software      -->
> <!-- distributed under the License is distributed on an "AS IS" BASIS,        -->
> <!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
> <!-- See the License for the specific language governing permissions and      -->
> <!-- limitations under the License.                                           -->
> <configuration>
> <property>
>   <name>fs.s3.buffer.dir</name>
>   <value>/mnt/s3,/mnt1/s3</value>
>   <final>true</final>
> </property>
> <property>
>   <name>fs.s3.impl</name>
>   <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
> </property>
> <property>
>   <name>fs.s3n.impl</name>
>   <value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value>
> </property>
>   <property>
>     <name>ipc.client.connect.max.retries.on.timeouts</name>
>     <value>5</value>
>   </property>
>   <property>
>     <name>hadoop.security.key.default.bitlength</name>
>     <value>256</value>
>   </property>
>   <property>
>     <name>hadoop.proxyuser.hadoop.groups</name>
>     <value>*</value>
>   </property>
>   <property>
>     <name>hadoop.tmp.dir</name>
>     <value>/mnt/var/lib/hadoop/tmp</value>
>   </property>
>   <property>
>     <name>hadoop.proxyuser.hadoop.hosts</name>
>     <value>*</value>
>   </property>
>   <property>
>     <name>io.file.buffer.size</name>
>     <value>65536</value>
>   </property>
>   <property>
>     <name>fs.AbstractFileSystem.s3.impl</name>
>     <value>org.apache.hadoop.fs.s3.EMRFSDelegate</value>
>   </property>
>   <property>
>     <name>fs.s3a.buffer.dir</name>
>     <value>/tmp</value>
>   </property>
>   <property>
>     <name>fs.s3bfs.impl</name>
>     <value>org.apache.hadoop.fs.s3.S3FileSystem</value>
>   </property>
> </configuration>
> {code}
> This bug is about excessive logging.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message