incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Williams <william...@gmail.com>
Subject Re: git commit: Let hdfs field manager ignore files he don't understand
Date Wed, 26 Feb 2014 01:43:39 GMT
On Tue, Feb 25, 2014 at 7:55 PM, Tim Williams <williamstw@gmail.com> wrote:
> On Tue, Feb 25, 2014 at 7:49 PM, Aaron McCurry <amccurry@gmail.com> wrote:
>> @@ -116,7 +117,7 @@ public class HdfsFieldManager extends BaseFieldManager {
>>        List<String> fieldNames = new ArrayList<String>();
>>        for (FileStatus fileStatus : listStatus) {
>>          if (!fileStatus.isDir()) {
>> -          fieldNames.add(fileStatus.getPath().getName());
>> +          fieldNames.add(fileStatus.getPath().getName().replace(TYPE_FILE_EXT,
>> ""));
>>          }
>>        }
>>
>> Do you think that this will cause problems for fields that are <some
>> family>.type<any suffix> ?  Maybe we should check that the file ends with
>> the TYPE_FILE_EXT and substring up to the beginning of the suffix?
>
> Ahh, literally have 'type' as the prefix of their column... yeah,
> that'd cause problems, nice catch.  I'll get a test for that one...

Interestingly enough, it actually still works just fine as is because
while we try to pull from the definitions map, we ultimately fall back
to storage anyway and since it's the correct fieldname coming in, it's
no problem.  I'll fix it, but I'm having a tough time crafting a test
that fails...

--tim

Mime
View raw message