spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Carman <>
Subject Spark Memory Utilities
Date Fri, 03 Apr 2015 15:45:24 GMT
I noticed spark has some nice memory tracking estimators in it, but they are private. We have
some custom implementations of RDD and PairRDD to suit our internal needs and it’d be fantastic
if we’d be able to just leverage the memory estimates that already exist in Spark.

Is there any change they can be made public inside the library or have some interface to them
such that children classes can make use of them?


Stephen Carman, M.S.
AI Engineer, Coldlight Solutions, LLC
Cell - 267 240 0363
This e-mail is intended solely for the above-mentioned recipient and it may contain confidential
or privileged information. If you have received it in error, please notify us immediately
and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance
on it. In addition, the contents of an attachment to this e-mail may contain software viruses
which could damage your own computer system. While ColdLight Solutions, LLC has taken every
reasonable precaution to minimize this risk, we cannot accept liability for any damage which
you sustain as a result of software viruses. You should perform your own virus checks before
opening the attachment.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message