Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Received-SPF: pass (athena.apache.org: domain of ewenstephan@gmail.com
 designates 209.85.223.173 as permitted sender)
MIME-Version: 1.0
Sender: ewenstephan@gmail.com
In-Reply-To: <6A9EAC02-3ED5-476E-BBFF-DD7AC3883461@icloud.com>
References: 
 <CANC1h_t-Z_86v39jGBAR_MNsvJ6wNymfJqRNa1EGervT9s541g@mail.gmail.com>
	<99df3a12-5bb2-4f9d-a6e2-d829cd7d37f0@me.com>
	<CANC1h_vnJvXxM9_EsvVbTFSr-Lwprwp4r_R5-CuqOFsqp+BEtg@mail.gmail.com>
	<53EDE7DD-24FC-454F-B320-57C36E2C9960@icloud.com>
	<ED914193-4BEB-4217-A85A-9D33BEA291BD@icloud.com>
	<CAC27z=MvkXHBG5gTToec6gPvZT=ikRj_Fan49H=5fHj4WATL6A@mail.gmail.com>
	<E89C8A1D-4E09-44A6-8844-E30AA7CBBC0F@icloud.com>
	<CAC27z=N6NFybmi7v1kPDsCeT6_u0_hAfXRfw9fKM9_ubWs_34g@mail.gmail.com>
	<AEBD44D0-9A16-49EE-91FB-79FCBF1B0B53@icloud.com>
	<CAC27z=PrQtwY_jkb-Bq78rhoVEnaPMrvSDcmW8JzQtjxZ6G37w@mail.gmail.com>
	<6A9EAC02-3ED5-476E-BBFF-DD7AC3883461@icloud.com>
Date: Thu, 5 Mar 2015 13:20:24 +0100
Message-ID: 
 <CANC1h_t+5S7sJ2Y7X1GCozU27YqVLrCmMJ=RmgmY9CjKVWSZdw@mail.gmail.com>
Subject: Re: Could not build up connection to JobManager
From: Stephan Ewen <sewen@apache.org>
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a113f99b4a4be990510899714

--001a113f99b4a4be990510899714
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Dulaj!

Okay, the logs give us some insight. Both setups seem to look good in terms
of TaskManager and JobManager startup.

In one of the logs (127.0.0.1) you submit a job. The job fails because the
TaskManager cannot grab the JAR file from the JobManager.
I think the problem is that the BLOB server binds to 0.0.0.0 - it should
bind to the same address as the JobManager actor system.

That should definitely be changed...

On Thu, Mar 5, 2015 at 10:08 AM, Dulaj Viduranga <vidura.me@icloud.com>
wrote:

> Hi,
> This is the log with setting =E2=80=9Clocalhost=E2=80=9D
> flink-Vidura-jobmanager-localhost.log <
> https://gist.github.com/viduranga/e9d43521587697de3eb5#file-flink-vidura-=
jobmanager-localhost-log
> >
>
> And this is the log with setting =E2=80=9C127.0.0.1=E2=80=9D
> flink-Vidura-jobmanager-localhost.log <
> https://gist.github.com/viduranga/5af6b05f204e1f4b344f#file-flink-vidura-=
jobmanager-localhost-log
> >
>
> > On Mar 5, 2015, at 2:23 PM, Till Rohrmann <trohrmann@apache.org> wrote:
> >
> > What does the jobmanager log says? I think Stephan added some more
> logging
> > output which helps us to debug this problem.
> >
> > On Thu, Mar 5, 2015 at 9:36 AM, Dulaj Viduranga <vidura.me@icloud.com>
> > wrote:
> >
> >> Using start-locat.sh.
> >> I=E2=80=99m using the original config yaml. I also tried changing jobm=
anager
> >> address in config to =E2=80=9C127.0.0.1 but no luck. With my changes i=
t works
> ok.
> >> The conf file follows.
> >>
> >>
> >>
> #########################################################################=
#######
> >> #  Licensed to the Apache Software Foundation (ASF) under one
> >> #  or more contributor license agreements.  See the NOTICE file
> >> #  distributed with this work for additional information
> >> #  regarding copyright ownership.  The ASF licenses this file
> >> #  to you under the Apache License, Version 2.0 (the
> >> #  "License"); you may not use this file except in compliance
> >> #  with the License.  You may obtain a copy of the License at
> >> #
> >> #      http://www.apache.org/licenses/LICENSE-2.0
> >> #
> >> #  Unless required by applicable law or agreed to in writing, software
> >> #  distributed under the License is distributed on an "AS IS" BASIS,
> >> #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> >> #  See the License for the specific language governing permissions and
> >> # limitations under the License.
> >>
> >>
> #########################################################################=
#######
> >>
> >>
> >>
> >>
> #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >> # Common
> >>
> >>
> #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >>
> >> jobmanager.rpc.address: 127.0.0.1
> >>
> >> jobmanager.rpc.port: 6123
> >>
> >> jobmanager.heap.mb: 256
> >>
> >> taskmanager.heap.mb: 512
> >>
> >> taskmanager.numberOfTaskSlots: 1
> >>
> >> parallelization.degree.default: 1
> >>
> >>
> >>
> #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >> # Web Frontend
> >>
> >>
> #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >>
> >> # The port under which the web-based runtime monitor listens.
> >> # A value of -1 deactivates the web server.
> >>
> >> jobmanager.web.port: 8081
> >>
> >> # The port uder which the standalone web client
> >> # (for job upload and submit) listens.
> >>
> >> webclient.port: 8080
> >>
> >>
> >>
> #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >> # Advanced
> >>
> >>
> #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >>
> >> # The number of buffers for the network stack.
> >> #
> >> # taskmanager.network.numberOfBuffers: 2048
> >>
> >> # Directories for temporary files.
> >> #
> >> # Add a delimited list for multiple directories, using the system
> directory
> >> # delimiter (colon ':' on unix) or a comma, e.g.:
> >> #     /data1/tmp:/data2/tmp:/data3/tmp
> >> #
> >> # Note: Each directory entry is read from and written to by a differen=
t
> I/O
> >> # thread. You can include the same directory multiple times in order t=
o
> >> create
> >> # multiple I/O threads against that directory. This is for example
> >> relevant for
> >> # high-throughput RAIDs.
> >> #
> >> # If not specified, the system-specific Java temporary directory
> >> (java.io.tmpdir
> >> # property) is taken.
> >> #
> >> # taskmanager.tmp.dirs: /tmp
> >>
> >> # Path to the Hadoop configuration directory.
> >> #
> >> # This configuration is used when writing into HDFS. Unless specified
> >> otherwise,
> >> # HDFS file creation will use HDFS default settings with respect to
> >> block-size,
> >> # replication factor, etc.
> >> #
> >> # You can also directly specify the paths to hdfs-default.xml and
> >> hdfs-site.xml
> >> # via keys 'fs.hdfs.hdfsdefault' and 'fs.hdfs.hdfssite'.
> >> #
> >> # fs.hdfs.hadoopconf: /path/to/hadoop/conf/
> >>
> >>
> >>> On Mar 5, 2015, at 2:03 PM, Till Rohrmann <trohrmann@apache.org>
> wrote:
> >>>
> >>> How did you start the flink cluster? Using the start-local.sh, the
> >>> start-cluster.sh or starting the job manager and task managers
> >> individually
> >>> using taskmanager.sh/jobmanager.sh. Could you maybe post the
> >>> flink-conf.yaml file, you're using?
> >>>
> >>> With your changes, everything works, right?
> >>>
> >>> On Thu, Mar 5, 2015 at 8:55 AM, Dulaj Viduranga <vidura.me@icloud.com=
>
> >>> wrote:
> >>>
> >>>> Hi Till,
> >>>> I=E2=80=99m sorry. It doesn=E2=80=99t seem to solve the problem. The=
 taskmanager still
> >>>> tries a 10.0.0.0/8 IP.
> >>>>
> >>>> Best regards.
> >>>>
> >>>>> On Mar 5, 2015, at 1:00 PM, Till Rohrmann <till.rohrmann@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi Dulaj,
> >>>>>
> >>>>> I looked through your commit and noticed that the JobClient might n=
ot
> >> be
> >>>>> listening on the right network interface. Your commit seems to fix
> it.
> >> I
> >>>>> just want to understand the problem properly and therefore I opened=
 a
> >>>>> branch with a small change. Could you try out whether this change
> would
> >>>>> also fix your problem? You can find the code here [1]. Would be
> awesome
> >>>> if
> >>>>> you checked it out and let it run on your cluster setting. Thanks a
> lot
> >>>>> Dulaj!
> >>>>>
> >>>>> [1]
> >>>>>
> >>>>
> >>
> https://github.com/tillrohrmann/flink/tree/fixLocalFlinkMiniClusterJobCli=
ent
> >>>>>
> >>>>> On Thu, Mar 5, 2015 at 4:21 AM, Dulaj Viduranga <
> vidura.me@icloud.com>
> >>>>> wrote:
> >>>>>
> >>>>>> The every change in the commit b7da22a is not required but I thoug=
ht
> >>>> they
> >>>>>> are appropriate.
> >>>>>>
> >>>>>>> On Mar 5, 2015, at 8:11 AM, Dulaj Viduranga <vidura.me@icloud.com=
>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>> I found many other places =E2=80=9Clocalhost=E2=80=9D is hard cod=
ed. I changed them
> >> in
> >>>> a
> >>>>>> better way I think. I made a pull request. Please review. b7da22a =
<
> >>>>>>
> >>>>
> >>
> https://github.com/viduranga/flink/commit/b7da22a562d3da5a9be2657308c0f82=
e4e2f80cd
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Mar 4, 2015, at 8:17 PM, Stephan Ewen <sewen@apache.org>
> wrote:
> >>>>>>>>
> >>>>>>>> If I recall correctly, we only hardcode "localhost" in the local
> >> mini
> >>>>>>>> cluster - do you think it is problematic there as well?
> >>>>>>>>
> >>>>>>>> Have you found any other places?
> >>>>>>>>
> >>>>>>>> On Mon, Mar 2, 2015 at 10:26 AM, Dulaj Viduranga <
> >>>> vidura.me@icloud.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> In some places of the code, "localhost" is hard coded. When it =
is
> >>>>>> resolved
> >>>>>>>>> by the DNS, it is posible to be directed  to a different IP oth=
er
> >>>> than
> >>>>>>>>> 127.0.0.1 (like private range 10.0.0.0/8). I changed those
> places
> >> to
> >>>>>>>>> 127.0.0.1 and it works like a charm.
> >>>>>>>>> But hard coding 127.0.0.1 is not a good option because when the
> >>>>>> jobmanager
> >>>>>>>>> ip is changed, this becomes an issue again. I'm thinking of
> setting
> >>>>>>>>> jobmanager ip from the config.yaml to these places.
> >>>>>>>>> If you have a better idea on doing this with your experience,
> >> please
> >>>>>> let
> >>>>>>>>> me know.
> >>>>>>>>>
> >>>>>>>>> Best.
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

--001a113f99b4a4be990510899714--