flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Issue while creating connector
Date Mon, 17 Apr 2017 07:21:05 GMT
As the exceptions says your outputformat isn't serializable; is the 
carboncontext marked as transient?

On 17.04.2017 09:12, Sangeeta Gulia wrote:
> Hi Team,
>
> Please ignore the previous mail.
>
> I am trying to create a connector for carbondata. Currently i am working on
> creating OutputFormat for carbondata. For that i need to create a
> carboncontext which i initialise in configure method. Internally
> CarbonContext uses sparkContext.
>
> I am able to create carbonContext through my code but when it run statement
>                               "executionEnvironment.execute()
> it throws the below error:
>
> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
> Error translating node 'Data Sink
> "org.apache.flink.CarbonDataOutputFormat@76225435"
> : NONE [[ GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[
> LocalProperties [ordering=null, grouped=null, unique=null] ]]': Could not
> write the user code wrapper class org.apache.flink.api.common.
> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
> org.apache.spark.sql.hive.client.ClientWrapper
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:381)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:106)
> at org.apache.flink.optimizer.plan.SingleInputPlanNode.
> accept(SingleInputPlanNode.java:198)
> at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
> OptimizedPlan.java:128)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
> compileJobGraph(JobGraphGenerator.java:192)
> at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188)
> at org.apache.flink.api.java.LocalEnvironment.execute(
> LocalEnvironment.java:91)
> at org.apache.flink.api.java.ExecutionEnvironment.execute(
> ExecutionEnvironment.java:926)
> at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
> Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException:
> Could not write the user code wrapper class org.apache.flink.api.common.
> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
> org.apache.spark.sql.hive.client.ClientWrapper
> at org.apache.flink.runtime.operators.util.TaskConfig.
> setStubWrapper(TaskConfig.java:281)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
> createDataSinkVertex(JobGraphGenerator.java:903)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:278)
> ... 8 more
> Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
> client.ClientWrapper
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at org.apache.flink.util.InstantiationUtil.serializeObject(
> InstantiationUtil.java:317)
> at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
> InstantiationUtil.java:254)
> at org.apache.flink.runtime.operators.util.TaskConfig.
> setStubWrapper(TaskConfig.java:279)
> ... 10 more
>
> Help me with the same.
>
> On Mon, Apr 17, 2017 at 12:39 PM, Sangeeta Gulia <sangeeta.gulia@knoldus.in>
> wrote:
>
>> Hi Team,
>>
>> I am trying to create a connector for carbondata. Currently i am working
>> on creating OutputFormat for carbondata. For that i need to create a
>> carboncontext which i initialise in configure method. Internally
>> CarbonContext uses sparkContext.
>>
>> I am able to create carbonContext through my code but when it run statement
>> "executionEnvironment.execute()
>>
>> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
>> Error translating node 'Data Sink "org.apache.flink.
>> CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties
>> [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null,
>> grouped=null, unique=null] ]]': Could not write the user code wrapper class
>> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io
>> .NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
>> JobGraphGenerator.java:381)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
>> JobGraphGenerator.java:106)
>> at org.apache.flink.optimizer.plan.SingleInputPlanNode.
>> accept(SingleInputPlanNode.java:198)
>> at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
>> OptimizedPlan.java:128)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
>> compileJobGraph(JobGraphGenerator.java:192)
>> at org.apache.flink.client.LocalExecutor.executePlan(
>> LocalExecutor.java:188)
>> at org.apache.flink.api.java.LocalEnvironment.execute(
>> LocalEnvironment.java:91)
>> at org.apache.flink.api.java.ExecutionEnvironment.execute(
>> ExecutionEnvironment.java:926)
>> at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
>> Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException:
>> Could not write the user code wrapper class org.apache.flink.api.common.
>> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
>> org.apache.spark.sql.hive.client.ClientWrapper
>> at org.apache.flink.runtime.operators.util.TaskConfig.
>> setStubWrapper(TaskConfig.java:281)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
>> createDataSinkVertex(JobGraphGenerator.java:903)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
>> JobGraphGenerator.java:278)
>> ... 8 more
>> Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
>> client.ClientWrapper
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>> at java.io.ObjectOutputStream.defaultWriteFields(
>> ObjectOutputStream.java:1548)
>> at java.io.ObjectOutputStream.writeSerialData(
>> ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>> ObjectOutputStream.java:1432)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>> at java.io.ObjectOutputStream.defaultWriteFields(
>> ObjectOutputStream.java:1548)
>> at java.io.ObjectOutputStream.writeSerialData(
>> ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>> ObjectOutputStream.java:1432)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>> at java.io.ObjectOutputStream.defaultWriteFields(
>> ObjectOutputStream.java:1548)
>> at java.io.ObjectOutputStream.writeSerialData(
>> ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>> ObjectOutputStream.java:1432)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
>> at org.apache.flink.util.InstantiationUtil.serializeObject(
>> InstantiationUtil.java:317)
>> at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
>> InstantiationUtil.java:254)
>> at org.apache.flink.runtime.operators.util.TaskConfig.
>> setStubWrapper(TaskConfig.java:279)
>> ... 10 more
>>
>>
>>
>> --
>> Warm Regards,
>>
>> Sangeeta Gulia
>> Software Consultant
>> Knoldus Software LLP
>>
>
>


Mime
View raw message