Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52ED610C4B for ; Tue, 13 Jan 2015 16:17:48 +0000 (UTC) Received: (qmail 45024 invoked by uid 500); 13 Jan 2015 16:17:49 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 44974 invoked by uid 500); 13 Jan 2015 16:17:49 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 44964 invoked by uid 99); 13 Jan 2015 16:17:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jan 2015 16:17:49 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric.newton@gmail.com designates 209.85.216.43 as permitted sender) Received: from [209.85.216.43] (HELO mail-qa0-f43.google.com) (209.85.216.43) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jan 2015 16:17:44 +0000 Received: by mail-qa0-f43.google.com with SMTP id v10so2757190qac.2 for ; Tue, 13 Jan 2015 08:17:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=3CIDcDl6ekE2pZgxG13Mpr476ll2G19b+oJA0px5ei4=; b=KyZoJ+lAMpb6yOnWOabNn5TX+AiH5vHi8eu3f8BS58R6v8lffCzLLCP2rIGub5HmSy XoCI1tlKPf9R8LKdQhryIomY8KGN5rFogYyuELW7hOv04Af6B+xI/1ymy/VRDU7ENbX2 +U6fWZAqujMi8WrseJgMZ6jsrcJBPk/Clho5280UoogMjPdt7wcBIa8WLOEyfvdBoAHq MPEmccUjN5pDm3tIa7KySZXux7+7AmH8V55QV2oJE8f0cYO2lKGjIbxkM1Oj0l8UtIHX gGpChRf1qo977GxS/L//NlfSZF+25MRlLvYgqTcc7FX5RYL/oQOGO4hEkPCqtXlc3rd7 JhJg== MIME-Version: 1.0 X-Received: by 10.224.113.200 with SMTP id b8mr21950476qaq.35.1421165842524; Tue, 13 Jan 2015 08:17:22 -0800 (PST) Received: by 10.96.139.69 with HTTP; Tue, 13 Jan 2015 08:17:22 -0800 (PST) In-Reply-To: References: Date: Tue, 13 Jan 2015 11:17:22 -0500 Message-ID: Subject: =?UTF-8?Q?Re=3A_=E5=9B=9E=E5=A4=8D=EF=BC=9Ahow_can_i_optimize_scan_speed_when_use_?= =?UTF-8?Q?batch_scan_=3F?= From: Eric Newton To: "user@accumulo.apache.org" Content-Type: multipart/alternative; boundary=047d7bea30cc35efc5050c8af518 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bea30cc35efc5050c8af518 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You should have received at least 1390 Key/Value pairs (#results=3D1390). If your application has many exact RowID look-ups, you may want to investigate Bloom filters. Consider turning on data block caching to reduce latency on future look-ups= . -Eric On Mon, Jan 12, 2015 at 8:15 PM, =E8=A6=83=E7=92=90 wr= ote: > i am sorry i do not know about the image. > > the log is this: > > > [17:50:38] TRACE > [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] > [org.apache.accumulo.core.util.OpTimer.start(OpTimer.java:39)] [21521] - > tid=3D65 oid=3D675 Continuing multi scan, scanid=3D-152589127623326551 > > [17:50:38] TRACE > [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] > [org.apache.accumulo.core.util.OpTimer.stop(OpTimer.java:49)] [21544] - > tid=3D65 oid=3D675 Got more multi scan results, #results=3D1390 > scanID=3D-152589127623326551 in 0.023 secs > > [17:50:38] TRACE > [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] > [org.apache.accumulo.core.util.OpTimer.start(OpTimer.java:39)] [21546] - > tid=3D65 oid=3D676 Continuing multi scan, scanid=3D-152589127623326551 > > [17:50:38] TRACE > [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] > [org.apache.accumulo.core.util.OpTimer.stop(OpTimer.java:49)] [21555] - > tid=3D45 oid=3D644 Got more multi scan results, #results=3D0 > scanID=3D-4477962012178388198 in 1.002 secs > > [17:50:38] TRACE > [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] > [org.apache.accumulo.core.util.OpTimer.start(OpTimer.java:39)] [21555] - > tid=3D45 oid=3D677 Continuing multi scan, scanid=3D-4477962012178388198 > > [17:50:38] TRACE > [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] > [org.apache.accumulo.core.util.OpTimer.stop(OpTimer.java:49)] [21596] - > tid=3D57 oid=3D645 Got more multi scan results, #results=3D0 > scanID=3D-8718025066902358141 in 1.003 secs > > [17:50:38] TRACE > [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] > [org.apache.accumulo.core.util.OpTimer.start(OpTimer.java:39)] [21596] - > tid=3D57 oid=3D678 Continuing multi scan, scanid=3D-8718025066902358141 > > > the scan spend long time but has no result. > > > i use 1.6.1,and the config output is this: > > > default | table.balancer ............................ | > org.apache.accumulo.server.master.balancer.DefaultLoadBalancer > > default | table.bloom.enabled ....................... | false > > default | table.bloom.error.rate .................... | 0.5% > > default | table.bloom.hash.type ..................... | murmur > > default | table.bloom.key.functor ................... | > org.apache.accumulo.core.file.keyfunctor.RowFunctor > > default | table.bloom.load.threshold ................ | 1 > > default | table.bloom.size .......................... | 1048576 > > default | table.cache.block.enable .................. | false > > default | table.cache.index.enable .................. | true > > default | table.classpath.context ................... | > > default | table.compaction.major.everything.idle .... | 1h > > default | table.compaction.major.ratio .............. | 3 > > default | table.compaction.minor.idle ............... | 5m > > default | table.compaction.minor.logs.threshold ..... | 3 > > table | table.constraint.1 ........................ | > org.apache.accumulo.core.constraints.DefaultKeySizeConstraint > > default | table.failures.ignore ..................... | false > > default | table.file.blocksize ...................... | 0B > > default | table.file.compress.blocksize ............. | 100K > > default | table.file.compress.blocksize.index ....... | 128K > > default | table.file.compress.type .................. | gz > > default | table.file.max ............................ | 15 > > default | table.file.replication .................... | 0 > > default | table.file.type ........................... | rf > > default | table.formatter ........................... | > org.apache.accumulo.core.util.format.DefaultFormatter > > default | table.groups.enabled ...................... | > > default | table.interepreter ........................ | > org.apache.accumulo.core.util.interpret.DefaultScanInterpreter > > table | table.iterator.majc.vers .................. | > 20,org.apache.accumulo.core.iterators.user.VersioningIterator > > table | table.iterator.majc.vers.opt.maxVersions .. | 1 > > table | table.iterator.minc.vers .................. | > 20,org.apache.accumulo.core.iterators.user.VersioningIterator > > table | table.iterator.minc.vers.opt.maxVersions .. | 1 > > table | table.iterator.scan.vers .................. | > 20,org.apache.accumulo.core.iterators.user.VersioningIterator > > table | table.iterator.scan.vers.opt.maxVersions .. | 1 > > default | table.majc.compaction.strategy ............ | > org.apache.accumulo.tserver.compaction.DefaultCompactionStrategy > > default | table.scan.max.memory ..................... | 512K > > default | table.security.scan.visibility.default .... | > > default | table.split.threshold ..................... | 1G > > default | table.walog.enabled ....................... | true > > > and my tablet server is 4 core,32G. > > > Thanks > > =E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6 > *=E5=8F=91=E4=BB=B6=E4=BA=BA:* Josh Elser > *=E6=94=B6=E4=BB=B6=E4=BA=BA:* user > *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4:* 2015=E5=B9=B41=E6=9C=8812=E6=97= =A5(=E5=91=A8=E4=B8=80)=E2=80=8723:52 > *=E4=B8=BB=E9=A2=98:* Re: =E5=9B=9E=E5=A4=8D=EF=BC=9Ahow can i optimize s= can speed when use batch scan ? > > FYI, images don't (typically) come across on the mailing list. Use some > external hosting and provide the link if it's important, please. > > How many tabletservers do you have? What version of Accumulo are you > running? Can you share the output of `config -t your_table_name`? > > Thanks. > > =E8=A6=83=E7=92=90 wrote: > > i look the trace log > > > > > > why it receive 0 result and spend so long? > > > > > > =E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6 > > *=E5=8F=91=E4=BB=B6=E4=BA=BA:* =E8=A6=83=E7=92=90 > > *=E6=94=B6=E4=BB=B6=E4=BA=BA:* user > > *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4:* 2015=E5=B9=B41=E6=9C=8812=E6=97= =A5(=E5=91=A8=E4=B8=80)=E2=80=8717:05 > > *=E4=B8=BB=E9=A2=98:* how can i optimize scan speed when use batch scan= ? > > > > hi all. > > > > now i have code like this: > > > > List rangeList=3D=E2=80=A6..; > > BatchScanner bs=3Dconn.createBatchScanner(); > > bs.setRanges(rangeList); > > > > > > the rangeList has many ranges about 1000,and every range has a random > > row id when i use Range.exact(new Text(=E2=80=A6)), > > but the speed is so slowly,it maybe spend 2-3s,how can i optimize it ? > > > > thanks > > --047d7bea30cc35efc5050c8af518 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You should have received at least 1390 Key/Value= pairs (#results=3D1390).

If your application has many exact R= owID look-ups, you may want to investigate Bloom filters.

Consider turning on data block caching to reduce latency on future look-up= s.

-Eric

On Mon, Jan 12, 2015 at 8:15 PM, =E8=A6=83=E7= =92=90 <luq.java@gmail.com> wrote:

i am sorry i do not know about the image.

the log is this:


[17:50:38] TRACE [org.apache.accu= mulo.core.client.impl.TabletServerBatchReaderIterator] [org.apache.accumulo= .core.util.OpTimer.start(OpTimer.java:39)] [21521] - tid=3D65 oid=3D675 =C2= =A0Continuing multi scan, scanid=3D-152589127623326551

[= 17:50:38] TRACE [org.apache.accumulo.core.client.impl.TabletServerBatchRead= erIterator] [org.apache.accumulo.core.util.OpTimer.stop(OpTimer.java:49)] [= 21544] - tid=3D65 oid=3D675 =C2=A0Got more multi scan results, #results=3D1= 390 =C2=A0scanID=3D-152589127623326551 in 0.023 secs

[17= :50:38] TRACE [org.apache.accumulo.core.client.impl.TabletServerBatchReader= Iterator] [org.apache.accumulo.core.util.OpTimer.start(OpTimer.java:39)] [2= 1546] - tid=3D65 oid=3D676 =C2=A0Continuing multi scan, scanid=3D-152589127= 623326551

[17:50:38] TRACE [org.apache.accumulo.core.cli= ent.impl.TabletServerBatchReaderIterator] [org.apache.accumulo.core.util.Op= Timer.stop(OpTimer.java:49)] [21555] - tid=3D45 oid=3D644 =C2=A0Got more mu= lti scan results, #results=3D0 =C2=A0scanID=3D-4477962012178388198 in 1.002= secs

[17:50:38] TRACE [org.apache.accumulo.core.client.= impl.TabletServerBatchReaderIterator] [org.apache.accumulo.core.util.OpTime= r.start(OpTimer.java:39)] [21555] - tid=3D45 oid=3D677 =C2=A0Continuing mul= ti scan, scanid=3D-4477962012178388198

[17:50:38] TRACE = [org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] [org= .apache.accumulo.core.util.OpTimer.stop(OpTimer.java:49)] [21596] - tid=3D5= 7 oid=3D645 =C2=A0Got more multi scan results, #results=3D0 =C2=A0scanID=3D= -8718025066902358141 in 1.003 secs

[17:50:38] TRACE [org= .apache.accumulo.core.client.impl.TabletServerBatchReaderIterator] [org.apa= che.accumulo.core.util.OpTimer.start(OpTimer.java:39)] [21596] - tid=3D57 o= id=3D678 =C2=A0Continuing multi scan, scanid=3D-8718025066902358141


the scan spend long time but has no res= ult.


i use 1.6.1,and the config = output is this:


default =C2=A0 = =C2=A0| table.balancer ............................ | org.apache.accumulo.s= erver.master.balancer.DefaultLoadBalancer

default =C2=A0= =C2=A0| table.bloom.enabled ....................... | false

default =C2=A0 =C2=A0| table.bloom.error.rate .................... | 0= .5%

default =C2=A0 =C2=A0| table.bloom.hash.type .......= .............. | murmur

default =C2=A0 =C2=A0| table.blo= om.key.functor ................... | org.apache.accumulo.core.file.keyfunct= or.RowFunctor

default =C2=A0 =C2=A0| table.bloom.load.th= reshold ................ | 1

default =C2=A0 =C2=A0| tabl= e.bloom.size .......................... | 1048576

defaul= t =C2=A0 =C2=A0| table.cache.block.enable .................. | false

default =C2=A0 =C2=A0| table.cache.index.enable ..............= .... | true

default =C2=A0 =C2=A0| table.classpath.conte= xt ................... |

default =C2=A0 =C2=A0| table.co= mpaction.major.everything.idle .... | 1h

default =C2=A0 = =C2=A0| table.compaction.major.ratio .............. | 3

= default =C2=A0 =C2=A0| table.compaction.minor.idle ............... | 5m

=

default =C2=A0 =C2=A0| table.compaction.minor.logs.threshol= d ..... | 3

table =C2=A0 =C2=A0 =C2=A0| table.constraint= .1 ........................ | org.apache.accumulo.core.constraints.DefaultK= eySizeConstraint

default =C2=A0 =C2=A0| table.failures.i= gnore ..................... | false

default =C2=A0 =C2= =A0| table.file.blocksize ...................... | 0B

de= fault =C2=A0 =C2=A0| table.file.compress.blocksize ............. | 100K

=

default =C2=A0 =C2=A0| table.file.compress.blocksize.index = ....... | 128K

default =C2=A0 =C2=A0| table.file.compres= s.type .................. | gz

default =C2=A0 =C2=A0| ta= ble.file.max ............................ | 15

default = =C2=A0 =C2=A0| table.file.replication .................... | 0

default =C2=A0 =C2=A0| table.file.type ........................... = | rf

default =C2=A0 =C2=A0| table.formatter ............= ............... | org.apache.accumulo.core.util.format.DefaultFormatter

=

default =C2=A0 =C2=A0| table.groups.enabled ...............= ....... |

default =C2=A0 =C2=A0| table.interepreter ....= .................... | org.apache.accumulo.core.util.interpret.DefaultScanI= nterpreter

table =C2=A0 =C2=A0 =C2=A0| table.iterator.ma= jc.vers .................. | 20,org.apache.accumulo.core.iterators.user.Ver= sioningIterator

table =C2=A0 =C2=A0 =C2=A0| table.iterat= or.majc.vers.opt.maxVersions .. | 1

table =C2=A0 =C2=A0 = =C2=A0| table.iterator.minc.vers .................. | 20,org.apache.accumul= o.core.iterators.user.VersioningIterator

table =C2=A0 = =C2=A0 =C2=A0| table.iterator.minc.vers.opt.maxVersions .. | 1

table =C2=A0 =C2=A0 =C2=A0| table.iterator.scan.vers ..............= .... | 20,org.apache.accumulo.core.iterators.user.VersioningIterator

table =C2=A0 =C2=A0 =C2=A0| table.iterator.scan.vers.opt.maxVe= rsions .. | 1

default =C2=A0 =C2=A0| table.majc.compacti= on.strategy ............ | org.apache.accumulo.tserver.compaction.DefaultCo= mpactionStrategy

default =C2=A0 =C2=A0| table.scan.max.m= emory ..................... | 512K

default =C2=A0 =C2=A0= | table.security.scan.visibility.default .... |

default = =C2=A0 =C2=A0| table.split.threshold ..................... | 1G

default =C2=A0 =C2=A0| table.walog.enabled ....................... = | true


and my tablet server is 4= core,32G.


Thanks


= =C2=A0=E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6=C2=A0<= span style=3D"color:#333;text-decoration:line-through;white-space:pre-wrap"= >
=E5=8F=91=E4=BB=B6=E4=BA=BA:=C2=A0Josh Elser<josh.elser@gmail.com<= /a>>
=E5=8F= =91=E9=80=81=E6=97=B6=E9=97=B4:=C2=A02015=E5=B9=B41=E6=9C=8812=E6=97=A5= (=E5=91=A8=E4=B8=80)=E2=80=8723:52
=E4=B8= =BB=E9=A2=98:=C2=A0Re: =E5=9B=9E=E5=A4=8D=EF=BC=9Ahow can i optimize sc= an speed when use batch scan ?

FYI, images don't (typically) come across on the mai=
ling list. Use some=20
external hosting and provide the link if it's important, please.

How many tabletservers do you have? What version of Accumulo are you=20
running? Can you share the output of `config -t your_table_name`?

Thanks.

=E8=A6=83=E7=92=90 wrote:
> i look the trace log
>
>
> why it receive 0 result and spend so long?
>
>
> =E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6
> *=E5=8F=91=E4=BB=B6=E4=BA=BA:* =E8=A6=83=E7=92=90<=
luq.java@gmail.com>
> *=E6=94=B6=E4=BB=B6=E4=BA=BA:* user<us=
er@accumulo.apache.org>
> *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4:* 2015=E5=B9=B41=E6=9C=8812=E6=
=97=A5(=E5=91=A8=E4=B8=80)=E2=80=8717:05
> *=E4=B8=BB=E9=A2=98:* how can i optimize scan speed when use batch sca=
n ?
>
> hi all.
>
> now i have code like this:
>
> List<Range> rangeList=3D=E2=80=A6..;
> BatchScanner bs=3Dconn.createBatchScanner();
> bs.setRanges(rangeList);
>
>
> the rangeList has many ranges about 1000,and every range has a random
> row id when i use Range.exact(new Text(=E2=80=A6)),
> but the speed is so slowly,it maybe spend 2-3s,how can i optimize it ?
>
> thanks

--047d7bea30cc35efc5050c8af518--