Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
From: Nico Kruber <nico@data-artisans.com>
To: Bernd.Winterstein@dev.helaba.de
Cc: user@flink.apache.org
Subject: Re: AW: Blob server not working with 1.4.0.RC2
Date: Wed, 06 Dec 2017 09:43:26 +0100
Message-ID: <65766799.vKMER10XJO@nico-pc.lan>
Organization: Data Artisans GmbH
In-Reply-To: <78E8C42DFD39B24F9D09C16D1CD84720A7BF29@hlsmx904.HLBENT.helaba.de>
References: <78E8C42DFD39B24F9D09C16D1CD84720A7BBEB@hlsmx904.HLBENT.helaba.de> <1682179253.1512.1512393418973.JavaMail.totemomail@hlb-stmail1.isp.fits> <78E8C42DFD39B24F9D09C16D1CD84720A7BF29@hlsmx904.HLBENT.helaba.de>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="nextPart4686580.VNLCTJa23e"; micalg="pgp-sha1"; protocol="application/pgp-signature"
archived-at: Wed, 06 Dec 2017 08:43:48 -0000

--nextPart4686580.VNLCTJa23e
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="iso-8859-1"

Hi Bernd,
at least from our side I don't see a change in the default BlobServer ports=
=20
between 1.3 and 1.4 - without configuration, the OS chooses the port.
If you want to influence the range it is chosen from (or want to fix a=20
specific port), you need to set the blob.server.port configuration paramete=
r=20
in Flink's flink-conf.yaml.


Regards
Nico

On Wednesday, 6 December 2017 08:45:19 CET Bernd.Winterstein@Dev.Helaba.de=
=20
wrote:
> Hi Nico
> I think there were changes in the default port fort the BLOB server. I
> missed the fact that the Kubernetes configuration was still exposing 6124
> for the JobManager BLOB server. Thanks
>=20
> Bernd
>=20
> -----Urspr=FCngliche Nachricht-----
> Von: Nico Kruber [mailto:nico@data-artisans.com]
> Gesendet: Montag, 4. Dezember 2017 14:17
> An: Winterstein, Bernd; user@flink.apache.org
> Betreff: Re: Blob server not working with 1.4.0.RC2
>=20
> Hi Bernd,
> thanks for the report. I tried to reproduce it locally but both a telnet
> connection to the BlobServer as well as the BLOB download by the
> TaskManagers work for me. Can you share your configuration that is causing
> the problem? You could also try increasing the log level to DEBUG and see
> if there is something more in the logs (the exception thrown in
> StaticFileServerHandler looks suspicious but is not related to the
> BlobServer). Apparently, the TaskManager resolves flink-jobmanager to
> 10.104.5.130. Is that the correct address and can the TaskManager talk to
> this IP? (may a firewall block this?)
>=20
> Did you, by any chance, set up SSL, too? There was a recent thread on the
> mailing list [1] where a had some problems with
> "security.ssl.verify-hostname" being set to true which may be related.
>=20
>=20
> Nico
>=20
> [1]
> https://lists.apache.org/thread.html/879d072bfd6761947b4bd703324489db50e8=
b14
> c328992118af875d8@%3Cuser.flink.apache.org%3E
> On 04/12/17 10:03, Bernd.Winterstein@Dev.Helaba.de wrote:
> > Hi
> > Since we switched to Release 1.4 the taskmanagers are unable to
> > download blobs from the jobmanager.
> > The taskmanager registration still works.
> > Netstat on jobmanager shows open ports at 6123 and 50000. But a telnet
> > connection from taskmanager to jobmanager on port 50000 times out.
> >=20
> > Any ideas are welcome.
> >=20
> > Regards
> >=20
> > Bernd
> >=20
> > Jobmanager log:
> >=20
> > 2017-12-04 08:48:30,167 INFO
> > org.apache.flink.runtime.jobmanager.JobManager                -
> > Starting JobManager actor
> > 2017-12-04 08:48:30,197 INFO
> > org.apache.flink.runtime.blob.BlobServer                      -
> > Created BLOB server storage directory
> > /tmp/blobStore-81cd12c7-394e-4777-85a1-98389b72dd08
> > 2017-12-04 08:48:30,205 INFO
> > org.apache.flink.runtime.blob.BlobServer                      -
> > Started BLOB server at 0.0.0.0:50000 - max concurrent requests: 50 -
> > max
> > backlog: 1000
> > 2017-12-04 08:48:30,608 INFO
> > org.apache.flink.runtime.jobmanager.JobManager                -
> > Starting JobManager at
> > akka.tcp://flink@flink-jobmanager:6123/user/jobmanager. 2017-12-04
> > 08:48:30,628 INFO
> > org.apache.flink.runtime.jobmanager.MemoryArchivist           -
> > Started memory archivist akka://flink/user/archive
> > 2017-12-04 08:48:30,676 INFO
> > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc
> > eManager
> > - Trying to associate with JobManager leader
> > akka.tcp://flink@flink-jobmanager:6123/user/jobmanager
> > 2017-12-04 08:48:30,692 INFO
> > org.apache.flink.runtime.jobmanager.JobManager                -
> > JobManager akka.tcp://flink@flink-jobmanager:6123/user/jobmanager was
> > granted leadership with leader session ID
> > Some(00000000-0000-0000-0000-000000000000).
> > 2017-12-04 08:48:30,700 INFO
> > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc
> > eManager
> > - Resource Manager associating with leading JobManager
> > Actor[akka://flink/user/jobmanager#886586058] - leader session
> > 00000000-0000-0000-0000-000000000000
> > 2017-12-04 08:53:50,635 INFO
> > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc
> > eManager
> > - TaskManager 627338086a766c140909ba45f2e717d0 has started.
> > 2017-12-04 08:53:50,638 INFO
> > org.apache.flink.runtime.instance.InstanceManager             -
> > Registered TaskManager at flink-taskmanager-65cf757d9b-hj65d
> > (akka.tcp://flink@flink-taskmanager-65cf757d9b-hj65d:45932/user/taskma
> > nager) as f9d2843d0223b15d8fce52aea8231cc6. Current number of
> > registered hosts is 1. Current number of alive task slots is 8.
> > 2017-12-04 08:53:50,658 WARN
> > akka.serialization.Serialization(akka://flink)                - Using
> > the default Java serializer for class
> > [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMes
> > sage] which is not recommended because of performance implications.
> > Use another serializer or disable this warning using the setting
> > 'akka.actor.warn-about-java-serializer-usage'
> > 2017-12-04 08:53:55,714 INFO
> > org.apache.flink.runtime.clusterframework.standalone.StandaloneResourc
> > eManager
> > - TaskManager 08c3e6f7c765e2ab88e2ea645049cb9d has started.
> > 2017-12-04 08:53:55,714 INFO
> > org.apache.flink.runtime.instance.InstanceManager             -
> > Registered TaskManager at flink-taskmanager-65cf757d9b-jtzw5
> > (akka.tcp://flink@flink-taskmanager-65cf757d9b-jtzw5:41710/user/taskma
> > nager) as da8a8da3650ce53f460784c54938a071. Current number of
> > registered hosts is 2. Current number of alive task slots is 16.
> > 2017-12-04 09:04:08,850 ERROR
> > org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHan
> > dler
> > - Caught exception
> > java.io.IOException: Operation timed out
> >=20
> >         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> >         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> >         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> >         at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> >         at
> >=20
> > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> >=20
> >=20
> > Taskmanager log:
> > 2017-12-04 08:53:55,511 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              -
> > Starting TaskManager actor at akka://flink/user/taskmanager#-977591027.
> > 2017-12-04 08:53:55,511 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              -
> > TaskManager data connection information:
> > 08c3e6f7c765e2ab88e2ea645049cb9d @ flink-taskmanager-65cf757d9b-jtzw5
> > (dataPort=3D42142)
> > 2017-12-04 08:53:55,512 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              -
> > TaskManager has 8 task slot(s).
> > 2017-12-04 08:53:55,513 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              - Memory
> > usage stats: [HEAP: 131/981/981 MB, NON HEAP: 34/35/-1 MB
> > (used/committed/max)]
> > 2017-12-04 08:53:55,518 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              - Trying
> > to register at JobManager
> > akka.tcp://flink@flink-jobmanager:6123/user/jobmanager (attempt 1,
> > timeout: 500 milliseconds)
> > 2017-12-04 08:53:55,671 WARN
> > akka.serialization.Serialization(akka://flink)                - Using
> > the default Java serializer for class
> > [org.apache.flink.runtime.messages.JobManagerMessages$LeaderSessionMes
> > sage] which is not recommended because of performance implications.
> > Use another serializer or disable this warning using the setting
> > 'akka.actor.warn-about-java-serializer-usage'
> > 2017-12-04 08:53:55,737 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              -
> > Successful registration at JobManager
> > (akka.tcp://flink@flink-jobmanager:6123/user/jobmanager), starting
> > network stack and library cache.
> > 2017-12-04 08:53:55,742 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              -
> > Determined BLOB server address to be
> > flink-jobmanager/10.104.5.130:50000. Starting BLOB cache.
> > 2017-12-04 08:53:55,752 INFO
> > org.apache.flink.runtime.blob.PermanentBlobCache              -
> > Created BLOB cache storage directory
> > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5
> > 2017-12-04 08:53:55,776 INFO
> > org.apache.flink.runtime.blob.TransientBlobCache              -
> > Created BLOB cache storage directory
> > /tmp/blobStore-8d7c8660-4455-43e9-93be-dadeb622b9e1
> > 2017-12-04 08:54:00,796 WARN
> > akka.serialization.Serialization(akka://flink)                - Using
> > the default Java serializer for class
> > [org.apache.flink.runtime.messages.TaskManagerMessages$Heartbeat]
> > which is not recommended because of performance implications. Use
> > another serializer or disable this warning using the setting
> > 'akka.actor.warn-about-java-serializer-usage'
> > 2017-12-04 09:10:38,726 WARN
> > akka.serialization.Serialization(akka://flink)                - Using
> > the default Java serializer for class
> > [org.apache.flink.runtime.metrics.dump.MetricDumpSerialization$MetricS
> > erializationResult] which is not recommended because of performance
> > implications. Use another serializer or disable this warning using the
> > setting 'akka.actor.warn-about-java-serializer-usage'
> > 2017-12-04 09:11:35,403 INFO
> > org.apache.flink.runtime.taskmanager.TaskManager              -
> > Received task Source: PACS008 Generator -> Double Submission Generator
> > -> Invalid IBAN Generator -> Sink: IP-PREPROCESSOR-INBOUND (1/1)
> > 2017-12-04 09:11:35,404 INFO
> > org.apache.flink.runtime.taskmanager.Task                     - Source:
> > PACS008 Generator -> Double Submission Generator -> Invalid IBAN
> > Generator -> Sink: IP-PREPROCESSOR-INBOUND (1/1)
> > (797de317bebf24df087f2da63cf5118e) switched from CREATED to DEPLOYING.
> > 2017-12-04 09:11:35,404 INFO
> > org.apache.flink.runtime.taskmanager.Task                     -
> > Creating FileSystem stream leak safety net for task Source: PACS008
> > Generator -> Double Submission Generator -> Invalid IBAN Generator ->
> > Sink:
> > IP-PREPROCESSOR-INBOUND (1/1) (797de317bebf24df087f2da63cf5118e)
> > [DEPLOYING]
> > 2017-12-04 09:11:35,408 WARN
> > akka.serialization.Serialization(akka://flink)                - Using
> > the default Java serializer for class
> > [org.apache.flink.runtime.messages.Acknowledge] which is not
> > recommended because of performance implications. Use another
> > serializer or disable this warning using the setting
> > 'akka.actor.warn-about-java-serializer-usage' 2017-12-04 09:11:35,410
> > INFO
> > org.apache.flink.runtime.taskmanager.Task                     -
> > Loading JAR files for task Source: PACS008 Generator -> Double
> > Submission Generator -> Invalid IBAN Generator -> Sink:
> > IP-PREPROCESSOR-INBOUND
> > (1/1) (797de317bebf24df087f2da63cf5118e) [DEPLOYING].
> > 2017-12-04 09:11:35,440 INFO
> > org.apache.flink.runtime.blob.BlobClient                      -
> > Downloading
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000
> > 2017-12-04 09:13:42,663 ERROR
> > org.apache.flink.runtime.blob.BlobClient                      - Failed
> > to fetch BLOB
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 and store it under
> > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000
> > 0000
> > Retrying...
> > 2017-12-04 09:13:42,663 INFO
> > org.apache.flink.runtime.blob.BlobClient                      -
> > Downloading
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 (retry 1)
> > 2017-12-04 09:15:49,895 ERROR
> > org.apache.flink.runtime.blob.BlobClient                      - Failed
> > to fetch BLOB
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 and store it under
> > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000
> > 0000
> > Retrying...
> > 2017-12-04 09:15:49,895 INFO
> > org.apache.flink.runtime.blob.BlobClient                      -
> > Downloading
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 (retry 2)
> > 2017-12-04 09:17:57,127 ERROR
> > org.apache.flink.runtime.blob.BlobClient                      - Failed
> > to fetch BLOB
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 and store it under
> > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000
> > 0000
> > Retrying...
> > 2017-12-04 09:17:57,127 INFO
> > org.apache.flink.runtime.blob.BlobClient                      -
> > Downloading
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 (retry 3)
> > 2017-12-04 09:20:04,359 ERROR
> > org.apache.flink.runtime.blob.BlobClient                      - Failed
> > to fetch BLOB
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 and store it under
> > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000
> > 0000
> > Retrying...
> > 2017-12-04 09:20:04,359 INFO
> > org.apache.flink.runtime.blob.BlobClient                      -
> > Downloading
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 (retry 4)
> > 2017-12-04 09:22:11,590 ERROR
> > org.apache.flink.runtime.blob.BlobClient                      - Failed
> > to fetch BLOB
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 and store it under
> > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000
> > 0000
> > Retrying...
> > 2017-12-04 09:22:11,591 INFO
> > org.apache.flink.runtime.blob.BlobClient                      -
> > Downloading
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 (retry 5)
> > 2017-12-04 09:24:18,823 ERROR
> > org.apache.flink.runtime.blob.BlobClient                      - Failed
> > to fetch BLOB
> > 27c97ae389eba8e2a4ec988cac86848a/p-756ea9bdf6f93f80dbe8b923ce288523120
> > d2b1b-392b55cacd4ed70b48e620f131a8bc61
> > from flink-jobmanager/10.104.5.130:50000 and store it under
> > /tmp/blobStore-39a9ec2f-32ff-4a31-a1e5-03294cd3cea5/incoming/temp-0000
> > 0000
> > No retries left.
> > java.io.IOException: Could not connect to BlobServer at address
> > flink-jobmanager/10.104.5.130:50000
> >=20
> >=20
> > _  ________________________________  _
> >=20
> >=20
> > Landesbank Hessen-Thueringen Girozentrale Anstalt des oeffentlichen
> > Rechts
> > Sitz: Frankfurt am Main / Erfurt
> > Amtsgericht Frankfurt am Main, HRA 29821 / Amtsgericht Jena, HRA
> > 102181
> >=20
> > Bitte nutzen Sie die E-Mail-Verbindung mit uns ausschliesslich zum
> > Informationsaustausch. Wir koennen auf diesem Wege keine
> > rechtsgeschaeftlichen Erklaerungen (Auftraege etc.) entgegennehmen.
> >=20
> > Der Inhalt dieser Nachricht ist vertraulich und nur fuer den
> > angegebenen Empfaenger bestimmt. Jede Form der Kenntnisnahme oder
> > Weitergabe durch Dritte ist unzulaessig. Sollte diese Nachricht nicht
> > fur Sie bestimmt sein, so bitten wir Sie, sich mit uns per E-Mail oder
> > telefonisch in Verbindung zu setzen.
> >=20
> > Please use your E-mail connection with us exclusively for the exchange
> > of information. We do not accept legally binding declarations (orders,
> > etc.) by this means of communication.
> >=20
> > The contents of this message is confidential and intended only for the
> > recipient indicated. Taking notice of this message or disclosure by
> > third parties is not permitted. In the event that this message is not
> > intended for you, please contact us via E-mail or phone.
>=20
> ________________________________
>=20
>=20
> Landesbank Hessen-Thueringen Girozentrale
> Anstalt des oeffentlichen Rechts
> Sitz: Frankfurt am Main / Erfurt
> Amtsgericht Frankfurt am Main, HRA 29821 / Amtsgericht Jena, HRA 102181
>=20
> Bitte nutzen Sie die E-Mail-Verbindung mit uns ausschliesslich zum
> Informationsaustausch. Wir koennen auf diesem Wege keine
> rechtsgeschaeftlichen Erklaerungen (Auftraege etc.) entgegennehmen.
>=20
> Der Inhalt dieser Nachricht ist vertraulich und nur fuer den angegebenen
> Empfaenger bestimmt. Jede Form der Kenntnisnahme oder Weitergabe durch
> Dritte ist unzulaessig. Sollte diese Nachricht nicht fur Sie bestimmt sei=
n,
> so bitten wir Sie, sich mit uns per E-Mail oder telefonisch in Verbindung
> zu setzen.
>=20
> Please use your E-mail connection with us exclusively for the exchange of
> information. We do not accept legally binding declarations (orders, etc.)
> by this means of communication.
>=20
> The contents of this message is confidential and intended only for the
> recipient indicated. Taking notice of this message or disclosure by third
> parties is not permitted. In the event that this message is not intended
> for you, please contact us via E-mail or phone.


--nextPart4686580.VNLCTJa23e
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part.
Content-Transfer-Encoding: 7Bit

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQTIh4KsbsNd3l7wd+cg8nJL2uqeWQUCWietrgAKCRAg8nJL2uqe
WdN+AKDAJMZe2bROZkrc/0DceCjNIVVFuwCgrQeviZljtI9SuKTBgl3BuvvmwN0=
=lgih
-----END PGP SIGNATURE-----

--nextPart4686580.VNLCTJa23e--