incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colton McInroy <col...@dosarrest.com>
Subject Re: Column Types
Date Tue, 05 Nov 2013 22:16:26 GMT
Looking at the other field types I have come up with this IP field type.

I am not sure what needs to be done to include it into the project 
though. Are there any other files that need to be modified to add the 
field type into the core or anything?

This field type should convert IP addresses into a long which can then 
be searched with a query specifying a range of IPs. Example...   
IP:[192.168.0.1 TO 192.168.0.255]

Thanks,
Colton McInroy

  * Director of Security Engineering

	
Phone
(Toll Free) 	
_US_ 	(888)-818-1344 Press 2
_UK_ 	0-800-635-0551 Press 2

My Extension 	101
24/7 Support 	support@dosarrest.com <mailto:support@dosarrest.com>
Email 	colton@dosarrest.com <mailto:colton@dosarrest.com>
Website 	http://www.dosarrest.com

On 10/21/2013 5:43 PM, Aaron McCurry wrote:
> Possibly, I will add an issue to take a look at it.
>
> https://issues.apache.org/jira/browse/BLUR-283
>
>
> On Mon, Oct 21, 2013 at 3:53 PM, Colton McInroy <colton@dosarrest.com>wrote:
>
>> Oh... strict is still showing up as false, even with -s -p strictTypes
>> true and adding a first mutate.
>>
>> Is this a bug?
>>
>>
>> Thanks,
>> Colton McInroy
>>
>>   * Director of Security Engineering
>>
>>
>> Phone
>> (Toll Free)
>> _US_    (888)-818-1344 Press 2
>> _UK_    0-800-635-0551 Press 2
>>
>> My Extension    101
>> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com>
>> Email   colton@dosarrest.com <mailto:colton@dosarrest.com>
>> Website         http://www.dosarrest.com
>>
>> On 10/21/2013 12:50 PM, Colton McInroy wrote:
>>
>>> I just tested, yes, that does appear to be the case...
>>>
>>> blur (default)> mutate Program_sshd 1 1 event Date:20131013124823
>>> blur (default)> schema Program_sshd
>>> Schema(table:Program_sshd, families:{event={Date=**ColumnDefinition(family:event,
>>> columnName:Date, subColumnName:null, fieldLessIndexed:false,
>>> fieldType:date, properties:{dateFormat=**yyyyMMddHHmmss})}})
>>> Program_sshd
>>> family : event
>>>          column   : Date
>>>                  fieldType : date
>>>                  prop      : dateFormat=yyyyMMddHHmmss
>>>
>>> Makes it kind of hard to look at the current schema before committing to
>>> start pushing data into it.
>>>
>>> Thanks,
>>> Colton McInroy
>>>
>>>   * Director of Security Engineering
>>>
>>>
>>> Phone
>>> (Toll Free)
>>> _US_     (888)-818-1344 Press 2
>>> _UK_     0-800-635-0551 Press 2
>>>
>>> My Extension     101
>>> 24/7 Support     support@dosarrest.com <mailto:support@dosarrest.com>
>>> Email     colton@dosarrest.com <mailto:colton@dosarrest.com>
>>> Website     http://www.dosarrest.com
>>>
>>> On 10/21/2013 12:28 PM, Aaron McCurry wrote:
>>>
>>>> Yes it might.  I will take a look this evening.
>>>>
>>>>
>>>> On Mon, Oct 21, 2013 at 3:20 PM, Colton McInroy <colton@dosarrest.com
>>>>> wrote:
>>>>   Hmmm... Is this expected? Does the schema only show up once I have added
>>>>> something? Also, strictTypes doesn't appear to be getting changed to
>>>>> true.
>>>>>
>>>>> blur (default)> create -c 10 -l file:///tmp/blur/tables/****Program_sshd
>>>>> -t
>>>>> Program_sshd -s
>>>>> blur (default)> definecolumn Program_sshd event Date date -p dateFormat
>>>>> yyyyMMddHHmmss
>>>>> blur (default)> schema Program_sshd
>>>>> Schema(table:Program_sshd, families:{})
>>>>> Program_sshd
>>>>> blur (default)> describe Program_sshd
>>>>> cluster                          : default
>>>>> name                             : Program_sshd
>>>>> enabled                          : true
>>>>> tableUri                         : file:///tmp/blur/tables/****
>>>>> Program_sshd
>>>>> shardCount                       : 10
>>>>> readOnly                         : false
>>>>> columnPreCache                   : null
>>>>> blockCaching                     : true
>>>>> blockCachingFileTypes            : null
>>>>> tableProperties                  : null
>>>>> strictTypes                      : false
>>>>> defaultMissingFieldType          : text
>>>>> defaultMissingFieldLessIndexin****g  : true
>>>>> defaultMissingFieldProps         : null
>>>>> blur (default)> disable Program_sshd
>>>>> blur (default)> remove Program_sshd
>>>>> blur (default)> create -c 10 -l file:///tmp/blur/tables/****Program_sshd
>>>>> -t
>>>>> Program_sshd -s -p strictTypes true
>>>>> blur (default)> describe Program_sshd
>>>>> cluster                          : default
>>>>> name                             : Program_sshd
>>>>> enabled                          : true
>>>>> tableUri                         : file:///tmp/blur/tables/****
>>>>> Program_sshd
>>>>> shardCount                       : 10
>>>>> readOnly                         : false
>>>>> columnPreCache                   : null
>>>>> blockCaching                     : true
>>>>> blockCachingFileTypes            : null
>>>>> tableProperties                  : null
>>>>> strictTypes                      : false
>>>>> defaultMissingFieldType          : text
>>>>> defaultMissingFieldLessIndexin****g  : true
>>>>> defaultMissingFieldProps         : null
>>>>> blur (default)> definecolumn Program_sshd event Date date -p dateFormat
>>>>> yyyyMMddHHmmss
>>>>> blur (default)> schema Program_sshd
>>>>> Schema(table:Program_sshd, families:{})
>>>>> Program_sshd
>>>>> blur (default)>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Colton McInroy
>>>>>
>>>>>    * Director of Security Engineering
>>>>>
>>>>>
>>>>> Phone
>>>>> (Toll Free)
>>>>> _US_    (888)-818-1344 Press 2
>>>>> _UK_    0-800-635-0551 Press 2
>>>>>
>>>>> My Extension    101
>>>>> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com>
>>>>> Email   colton@dosarrest.com <mailto:colton@dosarrest.com>
>>>>> Website         http://www.dosarrest.com
>>>>>
>>>>> On 10/21/2013 12:08 PM, Colton McInroy wrote:
>>>>>
>>>>>   Thanks,
>>>>>> Colton McInroy
>>>>>>
>>>>>>    * Director of Security Engineering
>>>>>>
>>>>>>
>>>>>> Phone
>>>>>> (Toll Free)
>>>>>> _US_     (888)-818-1344 Press 2
>>>>>> _UK_     0-800-635-0551 Press 2
>>>>>>
>>>>>> My Extension     101
>>>>>> 24/7 Support     support@dosarrest.com <mailto:support@dosarrest.com>
>>>>>> Email     colton@dosarrest.com <mailto:colton@dosarrest.com>
>>>>>> Website     http://www.dosarrest.com
>>>>>>
>>>>>> On 10/21/2013 11:41 AM, Aaron McCurry wrote:
>>>>>>
>>>>>>   Yes, however with the API changes that have/are being discussed
in
>>>>>>> others
>>>>>>> threads (Document vs. Record, Document Collection vs.Row, etc)
I want
>>>>>>> to
>>>>>>> change the value portion of the Column to have a Value type that
will
>>>>>>> be
>>>>>>> a
>>>>>>> union in Thrift instead of a struct.  This would allow us to
have all
>>>>>>> the
>>>>>>> basic types be defined in separate fields.  This stringValue
for
>>>>>>> string
>>>>>>> types, textValue for text types, intValue for int types, etc
that way
>>>>>>> when
>>>>>>> a table is not in strict mode it could better guess the correct
type
>>>>>>> instead of blindly choosing text.
>>>>>>>
>>>>>>>   Makes sense, for my purposes this should work well  enough
though. I
>>>>>> will
>>>>>> want strictTypes=true and define column types for the column I know
>>>>>> about,
>>>>>> and let automatic determination take care of the unspecified fields,
>>>>>> although that shouldn't be very many, since I should know all field
>>>>>> types
>>>>>> as I define them in my interface. Kinda sucks that they cannot be
>>>>>> changed
>>>>>> though, I'll have to build that into my GUI and make sure to take
note
>>>>>> of
>>>>>> that.
>>>>>>
>>>>>>   Ok so this code is to merely add the type to be available as a
type
>>>>>>> that
>>>>>>> can be used.  After this runs your new type will act just like
"text",
>>>>>>> "int", "date", or any other built in type.  After the type is
>>>>>>> registered
>>>>>>> in
>>>>>>> the system, either by table or system wide you will still need
to call
>>>>>>> definecolumn to make use of the new type.
>>>>>>>
>>>>>>> So in the example.
>>>>>>>
>>>>>>> tableDescriptor.****putToTableProperties("blur.****fieldtype.customfield1",
>>>>>>>
>>>>>>> "org.apache.blur.analysis.****type.ExampleType1");
>>>>>>>
>>>>>>> "blur.fieldtype." is the important part for the loader. The prefix
>>>>>>> tells
>>>>>>> the TableContext that this property is a new field type. So it
takes
>>>>>>> the
>>>>>>> value "org.apache.blur.analysis.****type.ExampleType1" for example
>>>>>>> and
>>>>>>> tries to
>>>>>>> load the class via the Class.forName method.  If successfully
and if
>>>>>>> it's a
>>>>>>> FieldTypeDefinition it will register is in the BaseFieldManager.
 Then
>>>>>>> the
>>>>>>> type is available for use.
>>>>>>>
>>>>>>> So "customfield1" is not even used.  It's only there to makes
the
>>>>>>> property
>>>>>>> be a unique name.
>>>>>>>
>>>>>>> Hope this helps.
>>>>>>>
>>>>>>>   Ahh... *sings* I can see clearly now the sun is out *sings*,
I was
>>>>>> misunderstanding what you meant first, that makes a lot more sense,
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>


Mime
View raw message