Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Message-ID: <1452897377.2787.17.camel@gmail.com>
Subject: Re: Custom ReadableSource implementations that reuse a PType?
From: Evan McClain <aeroevan@gmail.com>
To: dev@crunch.apache.org
Date: Fri, 15 Jan 2016 17:36:17 -0500
In-Reply-To: 
 <CANb5z2JChzHcHO279oFP+_YVkGZ9_C1pzffxCr23uoV6NED3ag@mail.gmail.com>
References: <1452791283.7336.16.camel@gmail.com>
	 <CANb5z2JChzHcHO279oFP+_YVkGZ9_C1pzffxCr23uoV6NED3ag@mail.gmail.com>
Content-Type: multipart/signed; micalg="pgp-sha256";
	protocol="application/pgp-signature"; boundary="=-CB5u2A7yk3K0omp0kNyn"
Mime-Version: 1.0

--=-CB5u2A7yk3K0omp0kNyn
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

The PCollection that is being cached is the result of a parallelDo (so
it is a DoCollection). The parent InputCollection has the source I
defined. Since the first read is triggered by that .cache() call on the
DoCollection, the file source given in the InputCollection is never
actually used.

Stepping through a debugger shows that even after using
pipeline.read(...).cache(), my ReadableSource.read() doesn't get called
even though I see my source in outputTargetsToMaterialize.value.source.
I only see AvroFileSourceTarget performing the reads so I'm probably
missing something (I'm new to crunch).

Any pointers would be appreciated.

Evan

On Thu, 2016-01-14 at 15:29 -0800, Josh Wills wrote:
> Hey Evan,
>=20
> So I must be missing some context here-- it looks to me like
> MRPipeline.getMaterializeSourceTarget first checks to see if you
> passed in
> an input collection type and then looks at its Source to see if it
> implements ReadableSource and uses that if it's available before
> defaulting
> to the ptype's default file source later on in the file via
> createIntermediateOutput. Is the PCollection you're trying to
> materialize
> not an input collection? e.g., is it an input collection that has had
> parallelDo called it on one or more times?
>=20
> J
>=20
> On Thu, Jan 14, 2016 at 9:08 AM, Evan McClain <aeroevan@gmail.com>
> wrote:
>=20
> > Hi list,
> >=20
> > I am trying to implement a custom ReadableSource by extending
> > FileSourceImpl, but it doesn't look like my source is actually
> > being
> > used since I am reusing AvroTypes (basically just trying to pass
> > through some avro getMeta() values).
> >=20
> > It looks like cache() -> materialize() ->
> > ptype.getDefaultFileSource()
> > is being used to perform the actual read (even though
> > pipeline.read()
> > is being passed my custom Source).
> >=20
> > Is there any way to do this without also implementing a custom
> > PType?
> >=20
> > Thanks
> > --
> > Evan McClain
> > https://keybase.io/aeroevan
> >=20
--=20
Evan McClain
https://keybase.io/aeroevan

--=-CB5u2A7yk3K0omp0kNyn
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAABCAAGBQJWmXRiAAoJEPQx78gG1JMfNrkH/2m1YfcQum1rf7jZkcX+Mgkn
MT5CLzpbwsg+Cnae828+oIVW885KY6Dau5ynldEX9VCv4zbO/nPoxt4l95YzlF4T
Q0psqXeQ7JIf21rOuZjzhQmIx/rdzA3SAcARDAIIKtRdHygOLpA1dmZxlt8FfARq
lYAR6Ocq4cbcCxUk6CLPvw/6r/hUUVrolLJYiRTnJgB5fuBYMssKqVyg8ifMp0zF
mk03GtgHmFQgFbkaJgj+N4c9d/3CSNWGOzEgRdCnNLsKJ6VZhuA7J04yPIrluBNk
4UmVavU8nM74Rgkpgmta2K0MrifRGtR1YN2qmawWcbgq+qBaxzuJTkZ/PcB+aJ8=
=8GPQ
-----END PGP SIGNATURE-----

--=-CB5u2A7yk3K0omp0kNyn--