spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang_intel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5682) Add encrypted shuffle in spark
Date Mon, 23 Mar 2015 05:49:11 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375430#comment-14375430
] 

liyunzhang_intel commented on SPARK-5682:
-----------------------------------------

Hi all:I have a question:is there any api in spark like getInstance(className:String):AnyRef
? I saw org.apache.spark.sql.hive .thriftserver.ReflectionUtils.scala, but not provide getInstance
function in it. 
Now I wrote a function getInstance:
[org.apache.spark.crypto.CryptoCodec#getInstance|https://github.com/kellyzly/spark/blob/8a74eea7d926507242c50b28c56962b1f1db256a/core/src/main/scala/org/apache/spark/crypto/CryptoCodec.scala/#l49]:
in my getInstance(className:String), i judge classname with "JceAesCtrCryptoCodec" and "OpensslAesCtrCryptoCodec"
and if the name equals "JceAesCtrCryptoCodec", it creates the instance by scala.reflect.runtime.universe
api. The code can be better like following way but I do not know how to write it. If some
knows, please tell me.
{code}
   def getInstance1(className:String):AnyRef={
       val m = universe.runtimeMirror(getClass.getClassLoader)
       var classLoader: ClassLoader = Thread.currentThread.getContextClassLoader
       val aClass:Class[_] =   Class.forName(className, true, classLoader)
       val aType: scala.reflect.api.TypeTags.TypeTag =  // how to write this line?
       val classCryptoCodec = universe.typeOf[aType]
         .typeSymbol.asClass
       val cm = m.reflectClass(classCryptoCodec)
       val ctor = universe.typeOf[aType].declaration(
         universe.nme.CONSTRUCTOR).asMethod
       val ctorm = cm.reflectConstructor(ctor)
       val p = ctorm()
       p
     }
{code}


> Add encrypted shuffle in spark
> ------------------------------
>
>                 Key: SPARK-5682
>                 URL: https://issues.apache.org/jira/browse/SPARK-5682
>             Project: Spark
>          Issue Type: New Feature
>          Components: Shuffle
>            Reporter: liyunzhang_intel
>         Attachments: Design Document of Encrypted Spark Shuffle_20150209.docx, Design
Document of Encrypted Spark Shuffle_20150318.docx
>
>
> Encrypted shuffle is enabled in hadoop 2.6 which make the process of shuffle data safer.
This feature is necessary in spark. We reuse hadoop encrypted shuffle feature to spark and
because ugi credential info is necessary in encrypted shuffle, we first enable encrypted shuffle
on spark-on-yarn framework.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message