couchdb-erlang mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Cottlehuber <...@jsonified.com>
Subject debugging
Date Mon, 05 Nov 2012 08:11:49 GMT
Hi there,

I wanted to throw in a brief set of notes on debugging stuff I learned
about, some recently & some taken from the internets.

most of this is for running "couchdb -i" but you can also do this via
remsh if you set a cookie and either sname/name up. Let me know if
that could be covered too.

# tell a non-networked erlang vm (e.g. started by couchdb -i) to get connected

    net_kernel:start([<newname>, shortnames]).
    erlang:set_cookie(node(), 'yumyumyum').

now you can remsh or redbug to it. (what's redbug??) ….

# killing off a process (e.g. a view server) that you don't want
running but there's no API to allow you to do that. The erlang shotgun
is here!

    i(). %% look for the process name. You could do this nicer with
list comprehensions etc below.
    exit(list_to_pid("<0.114.0>"), false). %% boom!

# get a list of all running processes and then do evil stuff

- in this case we find out who is the memory hog, using v(-1) to refer
to the result of the last computation in the shell and keeping the
info around in Procs just in case we want to check it later.

    %% this trick via Geoff Cant / @archaelus
    Procs =  [ {Pid, element(2, process_info(Pid, memory))}
            || Pid <- erlang:processes() ].

     hd( lists:reverse( lists:keysort( 2, v(-1) ) ) ).

# trace all calls going through a particular module

NB this is pretty savage if you get it wrong, you can *easily* crash
the vm. redbug is usually a better option, but I don't see how to get
it to do this nifty trick:

   open_tracer(Module) ->
         dbg:tracer(),
         dbg:p(all, [call]),
         dbg:tpl(Module, [{'_', [], [{return_trace}]}]).

# redbug

If you're using erlang and you don't know about redbug you are doing it wrong!!

Install redbug as part of eper kit, build is straightforwards using
rebar, add it into your ebin path. Again I can provide more details on
that if needed.

https://github.com/massemanet/eper/blob/master/doc/redbug.txt

Let's do a simple example - tracing all calls to lists:max, from our shell:

(akai@akai)20> redbug:start("lists:max").
ok
(akai@akai)21> lists:max([100,900,300,100,720]).

08:57:40 <{erlang,apply,2}> {lists,max,[[100,900,300,100,720]]}
900

08:57:40 <{erlang,apply,2}> {lists,max,[[900,300,100,720],100]}

08:57:40 <{erlang,apply,2}> {lists,max,[[300,100,720],900]}

08:57:40 <{erlang,apply,2}> {lists,max,[[100,720],900]}

08:57:40 <{erlang,apply,2}> {lists,max,[[720],900]}

08:57:40 <{erlang,apply,2}> {lists,max,[[],900]}

cool! you can see your recursion in action.

Fancy,  how about doing that from a remote node?

switch to your shell & set up redbug to call from a normal prompt/terminal.

launch redbug from your terminal:

% redbug akai@akai 5000 10 "lists:max"

this tetls it to connect to erlang host akai@akai, with a timeout of 5
seconds (you'll likely need longer the first few times), and collect a
maximum of 10 debug traces before stopping. Trace only lists:max.

redbug also accepts cookies and a bunch of other self-explanatory options.

whip back quickly into your erl shell and up arrow to restart the
lists:max function.

Your terminal will show the same goodies.

Now imagine doing the same thing on a live system, tracing a specific
call through mochiweb layer, or something in couch docs storage
system. Woot!!!

redbug can also optionally provide the call stack, arguments and
results but that starts getting hefty. So then write the results out
to a file.

Why redbug?

- ease of use compared to writing tracers yourself
- excellent logging
- filtering by time and number of results tends to avoid the "oops my
real-time debugging killed your couches".

# stats inside the vm

A number of options - use a fancy tool like vmstats from mononcqc or
for a more hands-on view use entop.


# debugging erlang startup

sometime erlang barfs on startup. how to proceed?

erl -init_debug  -env ERL_LIB
/usr/local/Cellar/couchdb/1.2.0/lib/couchdb/erlang/lib -couch_ini

And then hand-start each application:

application:start(crypto).
application:start(couch).
…

Sometimes this helps find the issue. A working sample is here
https://friendpaste.com/6zett3aAHSafRNZbgfMBoD

I should try the last step broken down into subsidiary steps rather
than a single application.

Anybody got other tips/tricks to share?

A+
Dave

Mime
View raw message