incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: git commit: Let hdfs field manager ignore files he don't understand
Date Wed, 26 Feb 2014 01:58:49 GMT
Ok, strange.  I would have thought that the test would fail...


On Tue, Feb 25, 2014 at 8:53 PM, Tim Williams <williamstw@gmail.com> wrote:

> It actually still does, because ultimately we do a tryToLoad() which
> loads from storage based on the passed field name.  The best i've got
> is to create a new one, loadfromstorage, the check
> getFieldNames().contains(foo) - which fails.
>
> On Tue, Feb 25, 2014 at 8:49 PM, Aaron McCurry <amccurry@gmail.com> wrote:
> > To make it fail I believe that you will have to load a field with "type"
> in
> > the beginning of the name.  Close the field manager and open a second
> one.
> >  I think that the field with "type" as the prefix won't reload.
> >
> >
> > On Tue, Feb 25, 2014 at 8:43 PM, Tim Williams <williamstw@gmail.com>
> wrote:
> >
> >> On Tue, Feb 25, 2014 at 7:55 PM, Tim Williams <williamstw@gmail.com>
> >> wrote:
> >> > On Tue, Feb 25, 2014 at 7:49 PM, Aaron McCurry <amccurry@gmail.com>
> >> wrote:
> >> >> @@ -116,7 +117,7 @@ public class HdfsFieldManager extends
> >> BaseFieldManager {
> >> >>        List<String> fieldNames = new ArrayList<String>();
> >> >>        for (FileStatus fileStatus : listStatus) {
> >> >>          if (!fileStatus.isDir()) {
> >> >> -          fieldNames.add(fileStatus.getPath().getName());
> >> >> +
> >>  fieldNames.add(fileStatus.getPath().getName().replace(TYPE_FILE_EXT,
> >> >> ""));
> >> >>          }
> >> >>        }
> >> >>
> >> >> Do you think that this will cause problems for fields that are <some
> >> >> family>.type<any suffix> ?  Maybe we should check that the
file ends
> >> with
> >> >> the TYPE_FILE_EXT and substring up to the beginning of the suffix?
> >> >
> >> > Ahh, literally have 'type' as the prefix of their column... yeah,
> >> > that'd cause problems, nice catch.  I'll get a test for that one...
> >>
> >> Interestingly enough, it actually still works just fine as is because
> >> while we try to pull from the definitions map, we ultimately fall back
> >> to storage anyway and since it's the correct fieldname coming in, it's
> >> no problem.  I'll fix it, but I'm having a tough time crafting a test
> >> that fails...
> >>
> >> --tim
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message