incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colton McInroy <col...@dosarrest.com>
Subject Re: Column Types
Date Tue, 05 Nov 2013 23:15:52 GMT
Hmm... looks like this may require a new field type added by lucene 
which handles larger than 64 bit numerical values. Unless anyone has any 
suggestions to work around this, I am currently talking with lucene 
developers about dealing with large numerical values.

Thanks,
Colton McInroy

  * Director of Security Engineering

	
Phone
(Toll Free) 	
_US_ 	(888)-818-1344 Press 2
_UK_ 	0-800-635-0551 Press 2

My Extension 	101
24/7 Support 	support@dosarrest.com <mailto:support@dosarrest.com>
Email 	colton@dosarrest.com <mailto:colton@dosarrest.com>
Website 	http://www.dosarrest.com

On 11/5/2013 2:38 PM, Colton McInroy wrote:
> Hmm...
>
> ipv6 cannot be converted to a long, so it is a bit more complicated, 
> but I will start trying to figure it out right now because that is 
> definately something I would like to see as well.
>
> Thanks,
> Colton McInroy
>
>  * Director of Security Engineering
>
>
> Phone
> (Toll Free)
> _US_     (888)-818-1344 Press 2
> _UK_     0-800-635-0551 Press 2
>
> My Extension     101
> 24/7 Support     support@dosarrest.com <mailto:support@dosarrest.com>
> Email     colton@dosarrest.com <mailto:colton@dosarrest.com>
> Website     http://www.dosarrest.com
>
> On 11/5/2013 2:21 PM, Garrett Barton wrote:
>> Awesome man,
>>
>> Would it be possible to add in ipv6 support?
>>
>>
>>
>>
>> On Tue, Nov 5, 2013 at 5:16 PM, Colton McInroy <colton@dosarrest.com> 
>> wrote:
>>
>>>   Looking at the other field types I have come up with this IP field 
>>> type.
>>>
>>> I am not sure what needs to be done to include it into the project 
>>> though.
>>> Are there any other files that need to be modified to add the field 
>>> type
>>> into the core or anything?
>>>
>>> This field type should convert IP addresses into a long which can 
>>> then be
>>> searched with a query specifying a range of IPs. Example...
>>> IP:[192.168.0.1 TO 192.168.0.255]
>>>
>>>   Thanks,
>>> Colton McInroy
>>>
>>>     - Director of Security Engineering
>>>
>>>     Phone
>>> (Toll Free)   *US* (888)-818-1344 Press 2  *UK* 0-800-635-0551 Press 
>>> 2    My
>>> Extension  101  24/7 Support  support@dosarrest.com  Email
>>> colton@dosarrest.com  Website http://www.dosarrest.com
>>>     On 10/21/2013 5:43 PM, Aaron McCurry wrote:
>>>
>>> Possibly, I will add an issue to take a look at it.
>>> https://issues.apache.org/jira/browse/BLUR-283
>>>
>>>
>>> On Mon, Oct 21, 2013 at 3:53 PM, Colton McInroy 
>>> <colton@dosarrest.com> <colton@dosarrest.com>wrote:
>>>
>>>
>>>   Oh... strict is still showing up as false, even with -s -p 
>>> strictTypes
>>> true and adding a first mutate.
>>>
>>> Is this a bug?
>>>
>>>
>>> Thanks,
>>> Colton McInroy
>>>
>>>   * Director of Security Engineering
>>>
>>>
>>> Phone
>>> (Toll Free)
>>> _US_    (888)-818-1344 Press 2
>>> _UK_    0-800-635-0551 Press 2
>>>
>>> My Extension    101
>>> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com> 
>>> <support@dosarrest.com>
>>> Email   colton@dosarrest.com <mailto:colton@dosarrest.com> 
>>> <colton@dosarrest.com>
>>> Website         http://www.dosarrest.com
>>>
>>> On 10/21/2013 12:50 PM, Colton McInroy wrote:
>>>
>>>
>>>   I just tested, yes, that does appear to be the case...
>>>
>>> blur (default)> mutate Program_sshd 1 1 event Date:20131013124823
>>> blur (default)> schema Program_sshd
>>> Schema(table:Program_sshd, 
>>> families:{event={Date=**ColumnDefinition(family:event,
>>> columnName:Date, subColumnName:null, fieldLessIndexed:false,
>>> fieldType:date, properties:{dateFormat=**yyyyMMddHHmmss})}})
>>> Program_sshd
>>> family : event
>>>          column   : Date
>>>                  fieldType : date
>>>                  prop      : dateFormat=yyyyMMddHHmmss
>>>
>>> Makes it kind of hard to look at the current schema before 
>>> committing to
>>> start pushing data into it.
>>>
>>> Thanks,
>>> Colton McInroy
>>>
>>>   * Director of Security Engineering
>>>
>>>
>>> Phone
>>> (Toll Free)
>>> _US_     (888)-818-1344 Press 2
>>> _UK_     0-800-635-0551 Press 2
>>>
>>> My Extension     101
>>> 24/7 Support     support@dosarrest.com 
>>> <mailto:support@dosarrest.com> <support@dosarrest.com>
>>> Email     colton@dosarrest.com <mailto:colton@dosarrest.com> 
>>> <colton@dosarrest.com>
>>> Website     http://www.dosarrest.com
>>>
>>> On 10/21/2013 12:28 PM, Aaron McCurry wrote:
>>>
>>>
>>>   Yes it might.  I will take a look this evening.
>>>
>>>
>>> On Mon, Oct 21, 2013 at 3:20 PM, Colton McInroy <colton@dosarrest.com
>>>
>>>   wrote:
>>>
>>>    Hmmm... Is this expected? Does the schema only show up once I 
>>> have added
>>>
>>>   something? Also, strictTypes doesn't appear to be getting changed to
>>> true.
>>>
>>> blur (default)> create -c 10 -l 
>>> file:///tmp/blur/tables/****Program_sshd
>>> -t
>>> Program_sshd -s
>>> blur (default)> definecolumn Program_sshd event Date date -p dateFormat
>>> yyyyMMddHHmmss
>>> blur (default)> schema Program_sshd
>>> Schema(table:Program_sshd, families:{})
>>> Program_sshd
>>> blur (default)> describe Program_sshd
>>> cluster                          : default
>>> name                             : Program_sshd
>>> enabled                          : true
>>> tableUri                         : file:///tmp/blur/tables/****
>>> Program_sshd
>>> shardCount                       : 10
>>> readOnly                         : false
>>> columnPreCache                   : null
>>> blockCaching                     : true
>>> blockCachingFileTypes            : null
>>> tableProperties                  : null
>>> strictTypes                      : false
>>> defaultMissingFieldType          : text
>>> defaultMissingFieldLessIndexin****g  : true
>>> defaultMissingFieldProps         : null
>>> blur (default)> disable Program_sshd
>>> blur (default)> remove Program_sshd
>>> blur (default)> create -c 10 -l 
>>> file:///tmp/blur/tables/****Program_sshd
>>> -t
>>> Program_sshd -s -p strictTypes true
>>> blur (default)> describe Program_sshd
>>> cluster                          : default
>>> name                             : Program_sshd
>>> enabled                          : true
>>> tableUri                         : file:///tmp/blur/tables/****
>>> Program_sshd
>>> shardCount                       : 10
>>> readOnly                         : false
>>> columnPreCache                   : null
>>> blockCaching                     : true
>>> blockCachingFileTypes            : null
>>> tableProperties                  : null
>>> strictTypes                      : false
>>> defaultMissingFieldType          : text
>>> defaultMissingFieldLessIndexin****g  : true
>>> defaultMissingFieldProps         : null
>>> blur (default)> definecolumn Program_sshd event Date date -p dateFormat
>>> yyyyMMddHHmmss
>>> blur (default)> schema Program_sshd
>>> Schema(table:Program_sshd, families:{})
>>> Program_sshd
>>> blur (default)>
>>>
>>>
>>>
>>>
>>> Thanks,
>>> Colton McInroy
>>>
>>>    * Director of Security Engineering
>>>
>>>
>>> Phone
>>> (Toll Free)
>>> _US_    (888)-818-1344 Press 2
>>> _UK_    0-800-635-0551 Press 2
>>>
>>> My Extension    101
>>> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com> 
>>> <support@dosarrest.com>
>>> Email   colton@dosarrest.com <mailto:colton@dosarrest.com> 
>>> <colton@dosarrest.com>
>>> Website         http://www.dosarrest.com
>>>
>>> On 10/21/2013 12:08 PM, Colton McInroy wrote:
>>>
>>>   Thanks,
>>>
>>>   Colton McInroy
>>>
>>>    * Director of Security Engineering
>>>
>>>
>>> Phone
>>> (Toll Free)
>>> _US_     (888)-818-1344 Press 2
>>> _UK_     0-800-635-0551 Press 2
>>>
>>> My Extension     101
>>> 24/7 Support     support@dosarrest.com 
>>> <mailto:support@dosarrest.com> <support@dosarrest.com>
>>> Email     colton@dosarrest.com <mailto:colton@dosarrest.com> 
>>> <colton@dosarrest.com>
>>> Website     http://www.dosarrest.com
>>>
>>> On 10/21/2013 11:41 AM, Aaron McCurry wrote:
>>>
>>>   Yes, however with the API changes that have/are being discussed in
>>>
>>>   others
>>> threads (Document vs. Record, Document Collection vs.Row, etc) I want
>>> to
>>> change the value portion of the Column to have a Value type that will
>>> be
>>> a
>>> union in Thrift instead of a struct.  This would allow us to have all
>>> the
>>> basic types be defined in separate fields.  This stringValue for
>>> string
>>> types, textValue for text types, intValue for int types, etc that way
>>> when
>>> a table is not in strict mode it could better guess the correct type
>>> instead of blindly choosing text.
>>>
>>>   Makes sense, for my purposes this should work well  enough though. I
>>>
>>>   will
>>> want strictTypes=true and define column types for the column I know
>>> about,
>>> and let automatic determination take care of the unspecified fields,
>>> although that shouldn't be very many, since I should know all field
>>> types
>>> as I define them in my interface. Kinda sucks that they cannot be
>>> changed
>>> though, I'll have to build that into my GUI and make sure to take note
>>> of
>>> that.
>>>
>>>   Ok so this code is to merely add the type to be available as a type
>>>
>>>   that
>>> can be used.  After this runs your new type will act just like "text",
>>> "int", "date", or any other built in type.  After the type is
>>> registered
>>> in
>>> the system, either by table or system wide you will still need to call
>>> definecolumn to make use of the new type.
>>>
>>> So in the example.
>>>
>>> tableDescriptor.****putToTableProperties("blur.****fieldtype.customfield1", 
>>>
>>>
>>> "org.apache.blur.analysis.****type.ExampleType1");
>>>
>>> "blur.fieldtype." is the important part for the loader. The prefix
>>> tells
>>> the TableContext that this property is a new field type. So it takes
>>> the
>>> value "org.apache.blur.analysis.****type.ExampleType1" for example
>>> and
>>> tries to
>>> load the class via the Class.forName method.  If successfully and if
>>> it's a
>>> FieldTypeDefinition it will register is in the BaseFieldManager.  Then
>>> the
>>> type is available for use.
>>>
>>> So "customfield1" is not even used.  It's only there to makes the
>>> property
>>> be a unique name.
>>>
>>> Hope this helps.
>>>
>>>   Ahh... *sings* I can see clearly now the sun is out *sings*, I was
>>>
>>>   misunderstanding what you meant first, that makes a lot more sense,
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message