Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Si-li Liu <unixipc@gmail.com>
Date: Mon, 7 Nov 2016 17:02:21 +0800
Message-ID: <CAD1b-Ri-P3Zf10Q7SLc7XoOSKRzWMZnJm=V9tFuJ=2ypDfr2QQ@mail.gmail.com>
Subject: Flink failed when can not connect to BlobServer
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a114ac30a2096ba0540b2485e
archived-at: Mon, 07 Nov 2016 09:02:27 -0000

--001a114ac30a2096ba0540b2485e
Content-Type: text/plain; charset=UTF-8

Hi, all

I use Flink DataSet API to do some batch job, read some log then group and
sort them. Our cluster has almost 2000 servers, we get used to use
traditional MR job, then I tried Flink to do some experiment job, but I
counter this error and can not continue, does anyone can help with it?

Our MR jobs also counter such connection error sometimes, but it will retry
serval times then get success. It seems that the whole calculation process
failed when one single task failed in Flink.

java.io.IOException: Cannot get library with hash
858478de9791c1a5fbbb138c02ec182b916f7962
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:262)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:116)
	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:721)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:472)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to fetch BLOB
858478de9791c1a5fbbb138c02ec182b916f7962 from /10.132.99.150:42927 and
store it under /tmp/blobStore-a2b79e70-74b9-49e8-a5bb-f2842aeec3b0/cache/blob_858478de9791c1a5fbbb138c02ec182b916f7962
	at org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:177)
	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:253)
	... 4 more
Caused by: java.io.IOException: Could not connect to BlobServer at
address /10.132.99.150:42927
	at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:88)
	at org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:124)
	... 5 more
Caused by: java.net.ConnectException: Connection timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:84)
	... 6 more


-- 
Best regards

Sili Liu

--001a114ac30a2096ba0540b2485e
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><span style=3D"font-size:14px">Hi, all</span><div style=3D=
"font-size:14px"><br></div><div style=3D"font-size:14px">I use Flink DataSe=
t API to do some batch job, read some log then group and sort them. Our clu=
ster has almost 2000 servers, we get used to use traditional MR job, then I=
 tried Flink to do some experiment job, but I counter this error and can no=
t continue, does anyone can help with it?</div><div style=3D"font-size:14px=
"><br></div><div style=3D"font-size:14px">Our MR jobs also counter such con=
nection error sometimes, but it will retry serval times then get success. I=
t seems that the whole calculation process failed when one single task fail=
ed in Flink.</div><div style=3D"font-size:14px"><br></div><div style=3D"fon=
t-size:14px"><pre class=3D"gmail-m_2908913124925131444exception gmail-m_290=
8913124925131444gmail-ng-binding" style=3D"white-space:pre-wrap;box-sizing:=
border-box;overflow:auto;font-family:menlo,monaco,consolas,&quot;courier ne=
w&quot;,monospace;font-size:13px;padding:0px;margin-top:0px;margin-bottom:0=
px;line-height:1.42857;word-break:break-all;word-wrap:break-word;color:rgb(=
51,51,51);background-color:transparent;border:none;border-radius:4px">java.=
io.IOException: Cannot get library with hash 858478de9791c1a5fbbb138c02ec18=
<wbr>2b916f7962
	at org.apache.flink.runtime.<wbr>execution.librarycache.<wbr>BlobLibraryCa=
cheManager.<wbr>registerReferenceToBlobKeyAndG<wbr>etURL(BlobLibraryCacheMa=
nager.<wbr>java:262)
	at org.apache.flink.runtime.<wbr>execution.librarycache.<wbr>BlobLibraryCa=
cheManager.<wbr>registerTask(<wbr>BlobLibraryCacheManager.java:<wbr>116)
	at org.apache.flink.runtime.<wbr>taskmanager.Task.<wbr>createUserCodeClass=
loader(<wbr>Task.java:721)
	at org.apache.flink.runtime.<wbr>taskmanager.Task.run(Task.<wbr>java:472)
	at java.lang.Thread.run(Thread.<wbr>java:745)
Caused by: java.io.IOException: Failed to fetch BLOB 858478de9791c1a5fbbb13=
8c02ec18<wbr>2b916f7962 from /<a href=3D"http://10.132.99.150:42927/" targe=
t=3D"_blank">10.132.99.150:42927</a> and store it under /tmp/blobStore-a2b7=
9e70-74b9-<wbr>49e8-a5bb-f2842aeec3b0/cache/<wbr>blob_<wbr>858478de9791c1a5=
fbbb138c02ec18<wbr>2b916f7962
	at org.apache.flink.runtime.blob.<wbr>BlobCache.getURL(BlobCache.<wbr>java=
:177)
	at org.apache.flink.runtime.<wbr>execution.librarycache.<wbr>BlobLibraryCa=
cheManager.<wbr>registerReferenceToBlobKeyAndG<wbr>etURL(BlobLibraryCacheMa=
nager.<wbr>java:253)
	... 4 more
Caused by: java.io.IOException: Could not connect to BlobServer at address =
/<a href=3D"http://10.132.99.150:42927/" target=3D"_blank">10.132.99.150:42=
927</a>
	at org.apache.flink.runtime.blob.<wbr>BlobClient.&lt;init&gt;(BlobClient.<=
wbr>java:88)
	at org.apache.flink.runtime.blob.<wbr>BlobCache.getURL(BlobCache.<wbr>java=
:124)
	... 5 more
Caused by: java.net.ConnectException: Connection timed out
	at java.net.PlainSocketImpl.<wbr>socketConnect(Native Method)
	at <a href=3D"http://java.net">java.net</a>.<wbr>AbstractPlainSocketImpl.<=
wbr>doConnect(<wbr>AbstractPlainSocketImpl.java:<wbr>350)
	at <a href=3D"http://java.net">java.net</a>.<wbr>AbstractPlainSocketImpl.<=
wbr>connectToAddress(<wbr>AbstractPlainSocketImpl.java:<wbr>206)
	at <a href=3D"http://java.net">java.net</a>.<wbr>AbstractPlainSocketImpl.<=
wbr>connect(<wbr>AbstractPlainSocketImpl.java:<wbr>188)
	at java.net.SocksSocketImpl.<wbr>connect(SocksSocketImpl.java:<wbr>392)
	at java.net.Socket.connect(<wbr>Socket.java:589)
	at java.net.Socket.connect(<wbr>Socket.java:538)
	at org.apache.flink.runtime.blob.<wbr>BlobClient.&lt;init&gt;(BlobClient.<=
wbr>java:84)
	... 6 more</pre></div><div><br></div>-- <br><div class=3D"gmail_signature"=
>Best regards<br>
<br>
Sili Liu</div>
</div>

--001a114ac30a2096ba0540b2485e--