accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shweta.agrawal" <shweta.agra...@orkash.com>
Subject Re: init method being called multiple times of WrappingIterator.
Date Thu, 02 Apr 2015 06:15:23 GMT
On Tuesday 31 March 2015 06:00 PM, shweta.agrawal wrote:
> On Monday 30 March 2015 08:03 PM, Josh Elser wrote:
>> Why are you using a print writer to get output from your iterator? 
>> Just use a logger and look in 
>> $ACCUMULO_HOME/logs/tserver_$hostname.debug.log (or wherever you 
>> configured logging). Create a log4j or slf4j Logger and use that 
>> instead of a print writer. (It's possible that your print writer is 
>> also what is slowing things down)
>>
>> In most real deployments, iterators should be faster on the server 
>> side than your client because you have N servers performing the work 
>> instead of your one client.
>>
>> It's not unheard of that a programming error is slowing down your 
>> iterator. Looking at what your iterator does (via logging) should 
>> help. Alternatively, you can use a remote debugger, connect a the 
>> tabletserver, and set breakpoints inside your iterator.
>>
>> shweta.agrawal wrote:
>>> On Monday 30 March 2015 09:58 AM, shweta.agrawal wrote:
>>>> Hi,
>>>>
>>>> Actually i am working on iterator, which i ran on server side by
>>>> making jar and also on client side on same data, but on server side
>>>> jar which i made is working slow than on client side. I am not able to
>>>> find what went wrong. is it possible to work same logic more fast on
>>>> client side than on accumulo iterators?
>>>>
>>>> time on client side:8s
>>>> time on server side:30s
>>>>
>>>> And to get the output i am writing output on text file through print
>>>> writer. To perform my task, i am calling my method on next method and
>>>> i am writing output to a file in next method. So actually i want to
>>>> know the final method which is called, so that i can write my output
>>>> to a file after performing all the task.
>>>>
>>>> Thanks and Regards
>>>> Shweta
>>>
>
> Hi,
>
> Without print writer also it is taking the same time. And i am trying 
> to use remote debugger as you suggested but i am facing problem.
>
> To enable remote debugger i changed this in accumulo-env.sh file:
> test -z "$ACCUMULO_TSERVER_OPTS" && export 
> ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx384m -Xms384m -Xdebug 
> -Xrunjdwp:transport=dt_socket,server=y,address=50095"
>
> But after changing this accumulo is not working. In terminal its 
> showing started and when i am going to accumulo shell its saying there 
> are no tablet servers. So please help me out in this. am i doing 
> something wrong?
>
> Monitor and tserver is not starting their logs are:
> Monitor Logs:
> 2015-03-31 17:36:09,516 [mortbay.log] INFO :  Logging to 
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
> org.mortbay.log.Slf4jLog
> 2015-03-31 17:36:09,535 [mortbay.log] INFO :  jetty-6.1.26
> 2015-03-31 17:36:09,607 [mortbay.log] WARN :  failed 
> SocketConnector@shweta:50095: java.net.BindException: Address already 
> in use
> 2015-03-31 17:36:09,608 [mortbay.log] WARN :  failed Server@6555694: 
> java.net.BindException: Address already in use
> 2015-03-31 17:36:09,608 [mortbay.log] INFO :  Stopped 
> SocketConnector@shweta:50095
>
> Tserver Logs:
> 2015-03-31 17:28:49,206 [tabletserver.TabletServer] INFO : unloaded 
> !0;~;!0<
> 2015-03-31 17:28:49,298 [tabletserver.TabletServer] INFO : unloaded !0<;~
> 2015-03-31 17:28:50,074 [tabletserver.TabletServer] INFO : unloaded 
> !0;!0<<
> 2015-03-31 17:28:50,121 [tabletserver.TabletServer] FATAL: Lost tablet 
> server lock (reason = LOCK_DELETED), exiting.
> 2015-03-31 17:28:50,122 [tabletserver.TabletServer] INFO : Master 
> requested tablet server halt
>
>
> Thanks and Regards
> Shweta
>
Hi,

Thanks for all your help. I got the logs from 
$ACCUMULO_HOME/logs/tserver_$hostname.debug.log. Upon analysing them and 
setting the iterator to work at Major compaction scope, I found out that 
the iterator speeds up and I was able to complete the computation in 887 
ms. So now I want to ask that why is there a difference in execution 
times when I run the same iterator at major compaction scope and scan 
scope? Also is there a way to detect the end of a Major Compaction 
programmatically?

Thanks and Regards
Shweta

Mime
View raw message