Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8503B2009FB for ; Fri, 6 May 2016 22:56:41 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 83A3F160A0C; Fri, 6 May 2016 20:56:41 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CDF621608F8 for ; Fri, 6 May 2016 22:56:40 +0200 (CEST) Received: (qmail 13596 invoked by uid 500); 6 May 2016 20:56:40 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 13585 invoked by uid 99); 6 May 2016 20:56:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2016 20:56:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 9CA591A02E5 for ; Fri, 6 May 2016 20:56:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 9RwVx_UdAncw for ; Fri, 6 May 2016 20:56:38 +0000 (UTC) Received: from mail-ob0-f172.google.com (mail-ob0-f172.google.com [209.85.214.172]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id CD9365F369 for ; Fri, 6 May 2016 20:56:37 +0000 (UTC) Received: by mail-ob0-f172.google.com with SMTP id dm5so57792933obc.1 for ; Fri, 06 May 2016 13:56:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Q6RFJ7e7jAMEBJag39TiL70FLXZlXz1B3zY5kbaEHvE=; b=Zo60VdKVkB56o0rPTUW38ANB9z+fiyDbh91p9a2izKHah3sXqX0Sim7TTumnRyd3mJ yNsxPBkrvmgymlxM/ZsGOngBUymrj/VdCpfUBPvYLxW8YxgWPRx6atvCNkTooxnAV6p+ VXWM+grc66691NnWJ+N86DOcNaP+udYXzTJROXg9RjpNSAafAWsnwzvs87KUQDWxYgRU aC2+zKCHRdoj4ylxqn/70Xi5ujNRWS94hRjRYVNOwXXBhw9kKTZSvrtzg9Wy9irArYwT n5gs34e0o8q/qrYeLAu+DOO0erDlUGdIcnclA5RSX/L4bNsgfok+akmRjX5QKzkqzb5q uXCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Q6RFJ7e7jAMEBJag39TiL70FLXZlXz1B3zY5kbaEHvE=; b=cB0AeMNtO/FsboyHpsodcN5qpLTgkOelpVg3HeGJKtzejoR7SXjIIvD8zJR8C3CETT 4JDiiDlAwIhPO8I4S6ZseEwhz1draKU7IgFaWMyuyj7D8wUGwWtZRypGOP5Vzm+dC4Vq +qiUA2+GjIAN2sG0jODPWpdD+3wpqFQ0rgFpROFn2xeluBz6Mw+iPaDO2F2dQ+aRwPLG lYBz8gRh/fNqv+3lPJivs4xLs6IG+cfBAii7Dct6pMVwEcLMH5M/4trUDQ5zYAC2JyQq faSbunzjRb2vBvMRd0aWameMLvbtStbQ8bfZdgucyOlesr9dc7i2YGcvZnZPA81XZMKs ZYTg== X-Gm-Message-State: AOPr4FWbhpdA/qdTGd58U1fQZGmvO2zlUVIFvDtMQtJlLTwDjENEuuxC0CNRam9b0oc3tVSH/1pIaFO86geWqg== X-Received: by 10.182.122.41 with SMTP id lp9mr10742296obb.16.1462568190658; Fri, 06 May 2016 13:56:30 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Josh Wills Date: Fri, 06 May 2016 20:56:19 +0000 Message-ID: Subject: Re: Materializing map of avros To: user Content-Type: multipart/alternative; boundary=bcaec539685676ce50053232b10c archived-at: Fri, 06 May 2016 20:56:41 -0000 --bcaec539685676ce50053232b10c Content-Type: text/plain; charset=UTF-8 Try using the ReadableData version of the PTable- it's an object that is serializable and you can read the data from it into whatever you want in the initialize method of the DoFn you pass it to. On Fri, May 6, 2016 at 1:03 PM David Ortiz wrote: > Hello, > > In attempt to make my code a little bit easier to following, I am > attempting to materialize a PTable to a map and then pass it into another > DoFn. Unfortunately, since the value is an Avro record, I am getting a > NotSerializableException out of the code when I try to use it. > > I attempting to get around this by converting the record into a > ByteBuffer with the avro utils, but lo and behold that's also not > Serializable. Since I do not see a convenient way to wrap a byte array > with crunch, has anyone had any luck with any other approaches to getting a > crunch-compatible serialized avro object? > > Thanks, > David Ortiz > --bcaec539685676ce50053232b10c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Try using the ReadableData version of the PTable- it's an object that i= s serializable and you can read the data from it into whatever you want in = the initialize method of the DoFn you pass it to.

On Fri, May 6, 2016 at 1:03 PM David Ortiz <dpo5003@gmail.com> wrote:
Hello,

=C2=A0 =C2=A0 =C2=A0 In attempt to make my code a little bit easier to fol= lowing, I am attempting to materialize a PTable to a map and then pass it i= nto another DoFn.=C2=A0 Unfortunately, since the value is an Avro record, I= am getting a NotSerializableException out of the code when I try to use it= .

=C2=A0 =C2=A0 =C2=A0I attempting to get around t= his by converting the record into a ByteBuffer with the avro utils, but lo = and behold that's also not Serializable.=C2=A0 Since I do not see a con= venient way to wrap a byte array with crunch, has anyone had any luck with = any other approaches to getting a crunch-compatible serialized avro object?=

Thanks,
=C2=A0 =C2=A0 =C2=A0David Ortiz=
--bcaec539685676ce50053232b10c--