incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthieu Morel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (S4-131) checkpoint error "CancellationException"
Date Wed, 27 Mar 2013 15:45:15 GMT

    [ https://issues.apache.org/jira/browse/S4-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615412#comment-13615412
] 

Matthieu Morel commented on S4-131:
-----------------------------------

Brilliant feedback, thanks!

I'm glad you found the issue, and the solution!


1. We need to update the documentation: we already indicate that requirements for a PE to
be checkpointed is to have serializable non-transient fields and empty no-args constructor,
but we should emphasize this, in particular the classical. Will do that as soon as possible.
 
2. In my opinion it is better to leave the default serializer as it is. Quoting Nathan Sweet
(Kryo's author) , "Objenesis isn't enabled by default. It should only be used for classes
designed to be used without calling a constructor or known to work when created that way,
so it should be a conscious decision to enable."  https://groups.google.com/d/msg/kryo-users/qAW435GGw-I/Nwi08R1p2s8J

Nevertheless, if you find it more convenient to use your proposed change, you might write
your own serializer that uses objenesis, write a module that injects this serializer in place
of the default one, and use it in the configuration of the app.


Since the problem is solved, the ticket should be closed in my opinion (marked as resolved,
unless I'm missing something).

Thanks!

                
> checkpoint error "CancellationException" 
> -----------------------------------------
>
>                 Key: S4-131
>                 URL: https://issues.apache.org/jira/browse/S4-131
>             Project: Apache S4
>          Issue Type: Bug
>    Affects Versions: 0.6
>         Environment: The configure like this:
> App program:
> ...
>  wordSumPE.setCheckpointingConfig(new CheckpointingConfig.Builder(CheckpointingMode.TIME).frequency(20).timeUnit(TimeUnit.SECONDS).build());
> ...
> delpoy:
> ./s4 deploy -a=example.wordcountApp -c=testCluster1 -appName=wordApp -p=s4.checkpointing.filesystem.storageRootPath=/home/tmp/s4checkpoint
-emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule
>            Reporter: Dingyu Yang 
>            Priority: Minor
>             Fix For: 0.7
>
>
> Then I get this error:
> 14:21:50.251 [Checkpointing-storage-0] WARN  org.apache.s4.core.ft.SaveStateTask - Cannot
save checkpoint : [PROTO_ID];[KEY] --> [example.WordSumPE];[word]
> java.util.concurrent.ExecutionException: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
> Serialization trace:
> classes (sun.misc.Launcher$AppClassLoader)
> contextClassLoader (java.lang.Thread)
> thread (java.util.concurrent.ThreadPoolExecutor$Worker)
> workers (java.util.concurrent.ThreadPoolExecutor)
> fetchingThreadPool (org.apache.s4.core.ft.SafeKeeper)
> checkpointingFramework (example.wordcountApp)
> app (org.apache.s4.core.Stream)
> downStream (example.WordSumPE)
>     at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) ~[na:1.6.0_22]
>     at java.util.concurrent.FutureTask.get(FutureTask.java:91) ~[na:1.6.0_22]
>     at org.apache.s4.core.ft.SaveStateTask.run(SaveStateTask.java:66) ~[bin/:na]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[na:1.6.0_22]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[na:1.6.0_22]
>     at java.lang.Thread.run(Thread.java:662) [na:1.6.0_22]
> Caused by: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
> Serialization trace:
> classes (sun.misc.Launcher$AppClassLoader)
> contextClassLoader (java.lang.Thread)
> thread (java.util.concurrent.ThreadPoolExecutor$Worker)
> workers (java.util.concurrent.ThreadPoolExecutor)
> fetchingThreadPool (org.apache.s4.core.ft.SafeKeeper)
> checkpointingFramework (example.wordcountApp)
> app (org.apache.s4.core.Stream)
> downStream (example.WordSumPE)
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:552) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:68)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:571) ~[kryo-2.20.jar:na]
>     at org.apache.s4.comm.serialize.KryoSerDeser.serialize(KryoSerDeser.java:91) ~[bin/:na]
>     at org.apache.s4.core.ProcessingElement.serializeState(ProcessingElement.java:802)
~[bin/:na]
>     at org.apache.s4.core.ft.SerializeTask.call(SerializeTask.java:42) ~[bin/:na]
>     at org.apache.s4.core.ft.SerializeTask.call(SerializeTask.java:1) ~[bin/:na]
>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) ~[na:1.6.0_22]
>     at java.util.concurrent.FutureTask.run(FutureTask.java:138) ~[na:1.6.0_22]
>     ... 3 common frames omitted
> Caused by: java.util.ConcurrentModificationException: null
>     at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) ~[na:1.6.0_22]
>     at java.util.AbstractList$Itr.next(AbstractList.java:343) ~[na:1.6.0_22]
>     at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:74)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:504) ~[kryo-2.20.jar:na]
>     at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
~[kryo-2.20.jar:na]
>     ... 35 common frames omitted
> I debug the program and at the position : SaveStateTask.run  "futureSerializedState.get(1000,
TimeUnit.MILLISECONDS)." The futureTask get "CancellationException" displayed above error.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message