spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ji Yan <ji...@drive.ai>
Subject Will Spark ever run the same task at the same time
Date Thu, 16 Feb 2017 18:34:14 GMT
Dear spark users,

Is there any mechanism in Spark that does not guarantee the idempotent
nature? For example, for stranglers, the framework might start another task
assuming the strangler is slow while the strangler is still running. This
would be annoying sometime when say the task is writing to a file, but have
the same tasks running at the same time may corrupt the file. From the
documentation page, I know that Spark's speculative execution mode is
turned off by default. Does anyone know any other mechanism in Spark that
may cause problem in scenario like this?

Thanks
Ji

-- 
 

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be unlawful.

Mime
View raw message