incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garrett Barton <garrett.bar...@gmail.com>
Subject Re: Column Types
Date Tue, 05 Nov 2013 23:58:14 GMT
How about converting the ipv4 to ipv6 and using whichever blur field-def
wraps keywordanalyzer on it?


On Tue, Nov 5, 2013 at 6:15 PM, Colton McInroy <colton@dosarrest.com> wrote:

> Hmm... looks like this may require a new field type added by lucene which
> handles larger than 64 bit numerical values. Unless anyone has any
> suggestions to work around this, I am currently talking with lucene
> developers about dealing with large numerical values.
>
>
> Thanks,
> Colton McInroy
>
>  * Director of Security Engineering
>
>
> Phone
> (Toll Free)
> _US_    (888)-818-1344 Press 2
> _UK_    0-800-635-0551 Press 2
>
> My Extension    101
> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com>
> Email   colton@dosarrest.com <mailto:colton@dosarrest.com>
> Website         http://www.dosarrest.com
>
> On 11/5/2013 2:38 PM, Colton McInroy wrote:
>
>> Hmm...
>>
>> ipv6 cannot be converted to a long, so it is a bit more complicated, but
>> I will start trying to figure it out right now because that is definately
>> something I would like to see as well.
>>
>> Thanks,
>> Colton McInroy
>>
>>  * Director of Security Engineering
>>
>>
>> Phone
>> (Toll Free)
>> _US_     (888)-818-1344 Press 2
>> _UK_     0-800-635-0551 Press 2
>>
>> My Extension     101
>> 24/7 Support     support@dosarrest.com <mailto:support@dosarrest.com>
>> Email     colton@dosarrest.com <mailto:colton@dosarrest.com>
>> Website     http://www.dosarrest.com
>>
>> On 11/5/2013 2:21 PM, Garrett Barton wrote:
>>
>>> Awesome man,
>>>
>>> Would it be possible to add in ipv6 support?
>>>
>>>
>>>
>>>
>>> On Tue, Nov 5, 2013 at 5:16 PM, Colton McInroy <colton@dosarrest.com>
>>> wrote:
>>>
>>>    Looking at the other field types I have come up with this IP field
>>>> type.
>>>>
>>>> I am not sure what needs to be done to include it into the project
>>>> though.
>>>> Are there any other files that need to be modified to add the field type
>>>> into the core or anything?
>>>>
>>>> This field type should convert IP addresses into a long which can then
>>>> be
>>>> searched with a query specifying a range of IPs. Example...
>>>> IP:[192.168.0.1 TO 192.168.0.255]
>>>>
>>>>   Thanks,
>>>> Colton McInroy
>>>>
>>>>     - Director of Security Engineering
>>>>
>>>>     Phone
>>>> (Toll Free)   *US* (888)-818-1344 Press 2  *UK* 0-800-635-0551 Press 2
>>>>    My
>>>> Extension  101  24/7 Support  support@dosarrest.com  Email
>>>> colton@dosarrest.com  Website http://www.dosarrest.com
>>>>     On 10/21/2013 5:43 PM, Aaron McCurry wrote:
>>>>
>>>> Possibly, I will add an issue to take a look at it.
>>>> https://issues.apache.org/jira/browse/BLUR-283
>>>>
>>>>
>>>> On Mon, Oct 21, 2013 at 3:53 PM, Colton McInroy <colton@dosarrest.com>
>>>> <colton@dosarrest.com>wrote:
>>>>
>>>>
>>>>   Oh... strict is still showing up as false, even with -s -p strictTypes
>>>> true and adding a first mutate.
>>>>
>>>> Is this a bug?
>>>>
>>>>
>>>> Thanks,
>>>> Colton McInroy
>>>>
>>>>   * Director of Security Engineering
>>>>
>>>>
>>>> Phone
>>>> (Toll Free)
>>>> _US_    (888)-818-1344 Press 2
>>>> _UK_    0-800-635-0551 Press 2
>>>>
>>>> My Extension    101
>>>> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com>
<
>>>> support@dosarrest.com>
>>>> Email   colton@dosarrest.com <mailto:colton@dosarrest.com> <
>>>> colton@dosarrest.com>
>>>> Website         http://www.dosarrest.com
>>>>
>>>> On 10/21/2013 12:50 PM, Colton McInroy wrote:
>>>>
>>>>
>>>>   I just tested, yes, that does appear to be the case...
>>>>
>>>> blur (default)> mutate Program_sshd 1 1 event Date:20131013124823
>>>> blur (default)> schema Program_sshd
>>>> Schema(table:Program_sshd, families:{event={Date=**
>>>> ColumnDefinition(family:event,
>>>> columnName:Date, subColumnName:null, fieldLessIndexed:false,
>>>> fieldType:date, properties:{dateFormat=**yyyyMMddHHmmss})}})
>>>> Program_sshd
>>>> family : event
>>>>          column   : Date
>>>>                  fieldType : date
>>>>                  prop      : dateFormat=yyyyMMddHHmmss
>>>>
>>>> Makes it kind of hard to look at the current schema before committing to
>>>> start pushing data into it.
>>>>
>>>> Thanks,
>>>> Colton McInroy
>>>>
>>>>   * Director of Security Engineering
>>>>
>>>>
>>>> Phone
>>>> (Toll Free)
>>>> _US_     (888)-818-1344 Press 2
>>>> _UK_     0-800-635-0551 Press 2
>>>>
>>>> My Extension     101
>>>> 24/7 Support     support@dosarrest.com <mailto:support@dosarrest.com>
<
>>>> support@dosarrest.com>
>>>> Email     colton@dosarrest.com <mailto:colton@dosarrest.com> <
>>>> colton@dosarrest.com>
>>>> Website     http://www.dosarrest.com
>>>>
>>>> On 10/21/2013 12:28 PM, Aaron McCurry wrote:
>>>>
>>>>
>>>>   Yes it might.  I will take a look this evening.
>>>>
>>>>
>>>> On Mon, Oct 21, 2013 at 3:20 PM, Colton McInroy <colton@dosarrest.com
>>>>
>>>>   wrote:
>>>>
>>>>    Hmmm... Is this expected? Does the schema only show up once I have
>>>> added
>>>>
>>>>   something? Also, strictTypes doesn't appear to be getting changed to
>>>> true.
>>>>
>>>> blur (default)> create -c 10 -l file:///tmp/blur/tables/****
>>>> Program_sshd
>>>> -t
>>>> Program_sshd -s
>>>> blur (default)> definecolumn Program_sshd event Date date -p dateFormat
>>>> yyyyMMddHHmmss
>>>> blur (default)> schema Program_sshd
>>>> Schema(table:Program_sshd, families:{})
>>>> Program_sshd
>>>> blur (default)> describe Program_sshd
>>>> cluster                          : default
>>>> name                             : Program_sshd
>>>> enabled                          : true
>>>> tableUri                         : file:///tmp/blur/tables/****
>>>> Program_sshd
>>>> shardCount                       : 10
>>>> readOnly                         : false
>>>> columnPreCache                   : null
>>>> blockCaching                     : true
>>>> blockCachingFileTypes            : null
>>>> tableProperties                  : null
>>>> strictTypes                      : false
>>>> defaultMissingFieldType          : text
>>>> defaultMissingFieldLessIndexin****g  : true
>>>> defaultMissingFieldProps         : null
>>>> blur (default)> disable Program_sshd
>>>> blur (default)> remove Program_sshd
>>>> blur (default)> create -c 10 -l file:///tmp/blur/tables/****
>>>> Program_sshd
>>>> -t
>>>> Program_sshd -s -p strictTypes true
>>>> blur (default)> describe Program_sshd
>>>> cluster                          : default
>>>> name                             : Program_sshd
>>>> enabled                          : true
>>>> tableUri                         : file:///tmp/blur/tables/****
>>>> Program_sshd
>>>> shardCount                       : 10
>>>> readOnly                         : false
>>>> columnPreCache                   : null
>>>> blockCaching                     : true
>>>> blockCachingFileTypes            : null
>>>> tableProperties                  : null
>>>> strictTypes                      : false
>>>> defaultMissingFieldType          : text
>>>> defaultMissingFieldLessIndexin****g  : true
>>>> defaultMissingFieldProps         : null
>>>> blur (default)> definecolumn Program_sshd event Date date -p dateFormat
>>>> yyyyMMddHHmmss
>>>> blur (default)> schema Program_sshd
>>>> Schema(table:Program_sshd, families:{})
>>>> Program_sshd
>>>> blur (default)>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Colton McInroy
>>>>
>>>>    * Director of Security Engineering
>>>>
>>>>
>>>> Phone
>>>> (Toll Free)
>>>> _US_    (888)-818-1344 Press 2
>>>> _UK_    0-800-635-0551 Press 2
>>>>
>>>> My Extension    101
>>>> 24/7 Support    support@dosarrest.com <mailto:support@dosarrest.com>
<
>>>> support@dosarrest.com>
>>>> Email   colton@dosarrest.com <mailto:colton@dosarrest.com> <
>>>> colton@dosarrest.com>
>>>> Website         http://www.dosarrest.com
>>>>
>>>> On 10/21/2013 12:08 PM, Colton McInroy wrote:
>>>>
>>>>   Thanks,
>>>>
>>>>   Colton McInroy
>>>>
>>>>    * Director of Security Engineering
>>>>
>>>>
>>>> Phone
>>>> (Toll Free)
>>>> _US_     (888)-818-1344 Press 2
>>>> _UK_     0-800-635-0551 Press 2
>>>>
>>>> My Extension     101
>>>> 24/7 Support     support@dosarrest.com <mailto:support@dosarrest.com>
<
>>>> support@dosarrest.com>
>>>> Email     colton@dosarrest.com <mailto:colton@dosarrest.com> <
>>>> colton@dosarrest.com>
>>>> Website     http://www.dosarrest.com
>>>>
>>>> On 10/21/2013 11:41 AM, Aaron McCurry wrote:
>>>>
>>>>   Yes, however with the API changes that have/are being discussed in
>>>>
>>>>   others
>>>> threads (Document vs. Record, Document Collection vs.Row, etc) I want
>>>> to
>>>> change the value portion of the Column to have a Value type that will
>>>> be
>>>> a
>>>> union in Thrift instead of a struct.  This would allow us to have all
>>>> the
>>>> basic types be defined in separate fields.  This stringValue for
>>>> string
>>>> types, textValue for text types, intValue for int types, etc that way
>>>> when
>>>> a table is not in strict mode it could better guess the correct type
>>>> instead of blindly choosing text.
>>>>
>>>>   Makes sense, for my purposes this should work well  enough though. I
>>>>
>>>>   will
>>>> want strictTypes=true and define column types for the column I know
>>>> about,
>>>> and let automatic determination take care of the unspecified fields,
>>>> although that shouldn't be very many, since I should know all field
>>>> types
>>>> as I define them in my interface. Kinda sucks that they cannot be
>>>> changed
>>>> though, I'll have to build that into my GUI and make sure to take note
>>>> of
>>>> that.
>>>>
>>>>   Ok so this code is to merely add the type to be available as a type
>>>>
>>>>   that
>>>> can be used.  After this runs your new type will act just like "text",
>>>> "int", "date", or any other built in type.  After the type is
>>>> registered
>>>> in
>>>> the system, either by table or system wide you will still need to call
>>>> definecolumn to make use of the new type.
>>>>
>>>> So in the example.
>>>>
>>>> tableDescriptor.****putToTableProperties("blur.****fieldtype.customfield1",
>>>>
>>>>
>>>> "org.apache.blur.analysis.****type.ExampleType1");
>>>>
>>>> "blur.fieldtype." is the important part for the loader. The prefix
>>>> tells
>>>> the TableContext that this property is a new field type. So it takes
>>>> the
>>>> value "org.apache.blur.analysis.****type.ExampleType1" for example
>>>> and
>>>> tries to
>>>> load the class via the Class.forName method.  If successfully and if
>>>> it's a
>>>> FieldTypeDefinition it will register is in the BaseFieldManager.  Then
>>>> the
>>>> type is available for use.
>>>>
>>>> So "customfield1" is not even used.  It's only there to makes the
>>>> property
>>>> be a unique name.
>>>>
>>>> Hope this helps.
>>>>
>>>>   Ahh... *sings* I can see clearly now the sun is out *sings*, I was
>>>>
>>>>   misunderstanding what you meant first, that makes a lot more sense,
>>>> Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message