spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Task failures and other problems
Date Thu, 09 Nov 2017 19:47:09 GMT
Maybe contact Oracle support?

Do you have maybe accidentally configured some firewall rules? Routing issues? Maybe only
one of the nodes...





> On 9. Nov 2017, at 20:04, Jan-Hendrik Zab <zab@l3s.de> wrote:
> 
> 
> Hello!
> 
> This might not be the perfect list for the issue, but I tried user@
> previously with the same issue, but with a bit less information to no
> avail.
> 
> So I'm hoping someone here can point me into the right direction.
> 
> We're using Spark 2.2 on CDH 5.13 (Hadoop 2.6 with patches) and a lot of
> our jobs fail, even when the jobs are super simple. For instance: [0]
> 
> We get two kinds of "errors", one where a task is actually marked as
> failed in the web ui [1]. Basically:
> 
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
> BP-1464899749-10.10.0.1-1378382027840:blk_1084837359_1099539407729
> file=/data/ia/derivatives/de/links/TA/part-68879.gz
> 
> See link for the stack trace.
> 
> When I check the block via "hdfs fsck -blockId blk_1084837359" all is
> well, I can also `-cat' the data into `wc'. It's a valid GZIP file.
> 
> The other kind of "error" we are getting are [2]:
> 
> DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 2648.1420920453265
msec.
> BlockReaderFactory: I/O error constructing remote block reader.
> java.net.SocketException: Network is unreachable
> DFSClient: Failed to connect to /10.12.1.26:50010 for block, add to
> deadNodes and continue. java.net.SocketException: Network is unreachable
> 
> These are logged in the stderr of _some_ of the executors.
> 
> I know that both things (at least to me) look more like a problem with
> HDFS and/or CDH. But we tried reading data via mapred jobs that
> essentially just manually opened the GZIP files, read them and printed
> some status info and those didn't produce any kind of error. The only
> thing we noticed was that sometimes the read() call apparently stalled
> for several minutes. But we couldn't identify a cause so far. And we
> also didn't see any errors in the CDH logs except maybe the following
> informational messages:
> 
> Likely the client has stopped reading, disconnecting it (node24.ib:50010:DataXceiver
error processing READ_BLOCK operation  src: /10.12.1.20:46518 dst: /10.12.1.24:50010); java.net.SocketTimeoutException:
120004 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected
local=/10.12.1.24:50010 remote=/10.12.1.20:46518]
> 
> All the systems (masters and nodes) can reach each other on the
> (infiniband) network. The systems communicate only over that one network
> (ie. datanodes only bind to one IP). /etc/hosts files are also the same
> on all systems and were distributed via ansible. But we also have a
> central DNS with the same data (and for PTR resolution) all systems are
> using.
> 
> The cluster has 37 nodes and 2 masters.
> 
> Suggestions are very welcome. :-)
> 
> [0] - http://www.l3s.de/~zab/link_converter.scala
> [1] - http://www.l3s.de/~zab/spark-errors-2.txt
> [2] - http://www.l3s.de/~zab/spark-errors.txt
> 
> Best,
>        -jhz
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message