crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-346) Don't deep-copy immutable Writable PTypes
Date Mon, 17 Feb 2014 17:06:19 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903374#comment-13903374
] 

Gabriel Reid commented on CRUNCH-346:
-------------------------------------

Thanks for catching the ByteBuffer issue [~stepinto].

About the PTypes#protos, agreed that the no-op deep copier should be used. The only issue
is the API with which it should be done. Right now the NoOpDeepCopier is used for all derived
types (i.e. everything created within PTypes), but that needs to be changed.

I'm was thinking about the possibility of adding a "derivedImmutable" method (in addition
to the normal "derived") method to allow creating PTypes for derived types that are immutable.
Any thoughts on that?

> Don't deep-copy immutable Writable PTypes
> -----------------------------------------
>
>                 Key: CRUNCH-346
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-346
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Chao Shi
>         Attachments: CRUNCH-346.patch
>
>
> I found getDetachedValue() appears quite often when jstack on one of my pipeline. A piece
of stacktrace is shown below. In the pipeline, most of types we used are immutable (e.g. java
primitives, strings, protobuf). I think we can avoid deep-copy overhead here.
> "main" prio=10 tid=0x00007f0de801d800 nid=0x7ef runnable [0x00007f0dee66c000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:52)
> 	at org.apache.hadoop.io.BytesWritable.<init>(BytesWritable.java:46)
> 	at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at java.lang.Class.newInstance0(Class.java:355)
> 	at java.lang.Class.newInstance(Class.java:308)
> 	at org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:63)
> 	at org.apache.crunch.types.writable.WritableDeepCopier.deepCopy(WritableDeepCopier.java:36)
> 	at org.apache.crunch.types.writable.WritableType.getDetachedValue(WritableType.java:125)
> 	at org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:54)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message