crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Shi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-346) Don't deep-copy immutable Writable PTypes
Date Mon, 17 Feb 2014 04:18:19 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902941#comment-13902941
] 

Chao Shi commented on CRUNCH-346:
---------------------------------

I think ByteBuffer is mutable and it directly reference to the buffer BytesWritable owns.
We can also make PTypes#protos use NoOpDeepCopier as protobuf message is immutable and all
internal data structures are copied from the original buffer (e.g ByteString). I'm not sure
about thrift message, but it seems to be mutable.

> Don't deep-copy immutable Writable PTypes
> -----------------------------------------
>
>                 Key: CRUNCH-346
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-346
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Chao Shi
>         Attachments: CRUNCH-346.patch
>
>
> I found getDetachedValue() appears quite often when jstack on one of my pipeline. A piece
of stacktrace is shown below. In the pipeline, most of types we used are immutable (e.g. java
primitives, strings, protobuf). I think we can avoid deep-copy overhead here.
> "main" prio=10 tid=0x00007f0de801d800 nid=0x7ef runnable [0x00007f0dee66c000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:52)
> 	at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:46)
> 	at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at java.lang.Class.newInstance0(Class.java:355)
> 	at java.lang.Class.newInstance(Class.java:308)
> 	at org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:63)
> 	at org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:36)
> 	at org.apache.crunch.types.writable.WritableType.getDetachedValue(WritableType.java:125)
> 	at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:54)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message