incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: custom fields with m/r
Date Wed, 13 Nov 2013 15:39:48 GMT
On Tue, Nov 12, 2013 at 12:41 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> If my memory is correct it should actually merge the system properties with
> td.properties before returning td
> On Nov 12, 2013 9:36 AM, "Tim Williams" <williamstw@gmail.com> wrote:
>
> > No joy.  I don't see how even that would work unless we're expecting
> > the system properties to be merged with the tabledescriptor.properties
> > when the tabledescriptor is returned?
>

I think this is what the expected behavior should be.


> >
> > To check that, I changed TableAdmin.describe locally to merge in the
> > system properties and it works fine.  I'm not sure if that's how you
> > were thinking it should behave or not though?  Thoughts?
>

Ok, we should probably modify the TableAdmin.describe to do that.  Perhaps
change TableAdmin.describe to pull it's TD from TableContext, that why the
properties will be merged in?  What do you think?

Aaron


> >
> > Thanks,
> > --tim
> >
> > On Tue, Nov 12, 2013 at 11:04 AM, Aaron McCurry <amccurry@gmail.com>
> > wrote:
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-blur.git;a=blob;f=blur-mapred/src/main/java/org/apache/blur/mapreduce/lib/BlurOutputFormat.java;h=caa350450918003175978924a9478533850eb7c3;hb=blur-console-v2#l397
> > >
> > > I think for the MR job to pickup the custom fields you will need to run
> > the
> > > following in order.
> > >
> > > TD td1 = new TD();
> > > client.create(td1);
> > > TD td2 = client.describe(td1.name);
> > > // Then if you use td2 for your MR job you will get the system fields I
> > > think.
> > >
> > > Let me know.
> > >
> > > Aaron
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Nov 12, 2013 at 10:54 AM, Tim Williams <williamstw@gmail.com>
> > wrote:
> > >
> > >> I don't see where the TableContext created in outputformat could have
> > >> access to the cluster's configuration or jar files.  Maybe when the
> > >> cluster is initialized it should write the configuration and jars to
> > >> HDFS, making it available to the output format?  If so, should the
> > >> writer be arranged with zk?  Or is there another plan I'm not seeing
> > >> in there?
> > >>
> > >> Thanks,
> > >> --tim
> > >>
> > >>
> > >> On Tue, Nov 12, 2013 at 10:34 AM, Aaron McCurry <amccurry@gmail.com>
> > >> wrote:
> > >> > It should be supported, this seems like a bug.
> > >> >
> > >> > Aaron
> > >> >
> > >> >
> > >> > On Tue, Nov 12, 2013 at 10:29 AM, Tim Williams <
> williamstw@gmail.com>
> > >> wrote:
> > >> >
> > >> >> It seems to be the case that when building a table with the
> > >> >> BlurOutputFormat, we need to follow the per-table custom field
> > >> >> definitions vs. cluster-wide.  If there's no plan to support
> > >> >> cluster-wide custom defs with m/r we should just document the
> > >> >> per-table version I reckon, thoughts?
> > >> >>
> > >> >> Thanks,
> > >> >> --tim
> > >> >>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message