accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Fetch Taking Longer Than Expected
Date Thu, 13 Aug 2015 15:52:54 GMT
Looks like we have a SetGroupsCommand (setgroups) which is analogous to 
the Java API in TableOperations, but we don't have those nice add/remove 
commands for locality groups (likely as they're harder to implement 
atomically).

Feel free to log a JIRA if you'd like to see add/remove instead of set. 
We can at least have the conversation on the matter -- there may be a 
reason I'm not aware of :)

Daniel Ruiz wrote:
> Thanks for the quick response.  It does make sense, although it would be
> nice to have an addgroup and delgroup (not sure if that exists in 1.7).
> This was extremely helpful, thanks again!
>
> V/r,
> -Daniel
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Wednesday, August 12, 2015 9:20 AM
> To: user@accumulo.apache.org
> Subject: Re: Fetch Taking Longer Than Expected
>
> Yup, that would be expected.
>
> Remember that doing `scan -c ...` is an unbounded search over your
> entire table. So, it takes approximately 3 minutes to read your
> GUIDIndexTable. Because you have a single locality group, all of the
> columns in your table are grouped together.
>
> One exercise that may be interesting for yourself is to create a
> locality group that has your specific column family in it, compact your
> GUIDIndexTable, and rerun your `scan -c` query. The speed should be
> similar to your exact scan. Removing the locality group and
> re-compacting the table should return the query time back to the slow 3
> minutes.
>
> Does that make sense?
>
> Daniel Ruiz wrote:
>> Hi All,
>>
>> I am having an issue where column fetches are taking over a minute on
>> 1.6.3. I don't believe this should be case and my experience in the past
>> supports the idea that fetches should be very fast.
>>
>> For example we doing a scan on the table gives results instantly but
>> doing a scan -c vesselmmsitext=2706758566 takes 2 minutes and 44 seconds
>> (plus or minus 1 second).
>>
>> Figure 1.1. Generated Test Data on GUIDIndexTable
>>
>> Here is the table config
>>
>>
> -----------+---------------------------------------------------------------+
> ----------------------------------------------------------------------------
> -----
>> SCOPE | NAME | VALUE
>>
>>
> -----------+---------------------------------------------------------------+
> ----------------------------------------------------------------------------
> -----
>> default | table.balancer ..............................................
>> | org.apache.accumulo.server.master.balancer.DefaultLoadBalancer
>>
>> default | table.bloom.enabled .........................................
>> | false
>>
>> default | table.bloom.error.rate ......................................
>> | 0.5%
>>
>> default | table.bloom.hash.type .......................................
>> | murmur
>>
>> default | table.bloom.key.functor .....................................
>> | org.apache.accumulo.core.file.keyfunctor.RowFunctor
>>
>> default | table.bloom.load.threshold .................................. |
> 1
>> default | table.bloom.size ............................................
>> | 1048576
>>
>> default | table.cache.block.enable ....................................
>> | false
>>
>> default | table.cache.index.enable ....................................
>> | true
>>
>> default | table.classpath.context ..................................... |
>>
>> default | table.compaction.major.everything.idle ...................... |
> 1h
>> default | table.compaction.major.ratio ................................ |
> 3
>> default | table.compaction.minor.idle ................................. |
> 5m
>> default | table.compaction.minor.logs.threshold ....................... |
> 3
>> table | table.constraint.1 .......................................... |
>> org.apache.accumulo.core.constraints.DefaultKeySizeConstraint
>>
>> default | table.failures.ignore .......................................
>> | false
>>
>> default | table.file.blocksize ........................................ |
> 0B
>> default | table.file.compress.blocksize ...............................
>> | 100K
>>
>> default | table.file.compress.blocksize.index .........................
>> | 128K
>>
>> default | table.file.compress.type .................................... |
> gz
>> default | table.file.max .............................................. |
> 15
>> default | table.file.replication ...................................... |
> 0
>> default | table.file.type ............................................. |
> rf
>> default | table.formatter .............................................
>> | org.apache.accumulo.core.util.format.DefaultFormatter
>>
>> default | table.groups.enabled ........................................ |
>>
>> default | table.interepreter ..........................................
>> | org.apache.accumulo.core.util.interpret.DefaultScanInterpreter
>>
>> table | table.iterator.majc.AgeOffIterator##GUIDIndexTable .......... |
>> 1,org.apache.accumulo.core.iterators.user.AgeOffFilter
>>
>> table | table.iterator.majc.AgeOffIterator##GUIDIndexTable.opt.ttl .. |
>> 2592000000
>>
>> table | table.iterator.majc.vers .................................... |
>> 20,org.apache.accumulo.core.iterators.user.VersioningIterator
>>
>> table | table.iterator.majc.vers.opt.maxVersions .................... | 1
>>
>> table | table.iterator.minc.AgeOffIterator##GUIDIndexTable .......... |
>> 1,org.apache.accumulo.core.iterators.user.AgeOffFilter
>>
>> table | table.iterator.minc.AgeOffIterator##GUIDIndexTable.opt.ttl .. |
>> 2592000000
>>
>> table | table.iterator.minc.vers .................................... |
>> 20,org.apache.accumulo.core.iterators.user.VersioningIterator
>>
>> table | table.iterator.minc.vers.opt.maxVersions .................... | 1
>>
>> table | table.iterator.scan.AgeOffIterator##GUIDIndexTable .......... |
>> 1,org.apache.accumulo.core.iterators.user.AgeOffFilter
>>
>> table | table.iterator.scan.AgeOffIterator##GUIDIndexTable.opt.ttl .. |
>> 2592000000
>>
>> ---------------------------------------------------------- hit any key
>> to continue or 'q' to quit
>> ----------------------------------------------------------
>>
>> table | table.iterator.scan.vers .................................... |
>> 20,org.apache.accumulo.core.iterators.user.VersioningIterator
>>
>> table | table.iterator.scan.vers.opt.maxVersions .................... | 1
>>
>> default | table.majc.compaction.strategy ..............................
>> | org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy
>>
>> default | table.scan.max.memory .......................................
>> | 512K
>>
>> table | @override ................................................ | 1M
>>
>> default | table.security.scan.visibility.default ...................... |
>>
>> default | table.split.threshold ....................................... |
> 1G
>> default | table.walog.enabled .........................................
>> | true
>>
>>
> -----------+---------------------------------------------------------------+
> ----------------------------------------------------------------------------
> -----
>> More Table Info:
>>
>> GUIDIndexTable<http://107.23.12.24:50095/tables?t=f>
>>
>> 	
>>
>> ONLINE
>>
>> 	
>>
>> 2
>>
>> 	
>>
>> 0
>>
>> 	
>>
>> 82.56M
>>
>> 	
>>
>> 810.00K
>>
>> 	
>>
>> 159
>>
>> Please let me know if I am doing something wrong to if there is more
>> information you need.
>>
>> V/r,
>>
>> -Daniel
>>
>

Mime
View raw message