couchdb-erlang mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From david martin <david.mar...@lymegreen.co.uk>
Subject Re: Learning Erlang/OTP and CouchDB internals together
Date Sun, 18 Nov 2012 22:35:43 GMT
On 16/11/12 19:05, Dave Cottlehuber wrote:
> Many thanks for all replies on my other thread!
>
> Erlang is amazing, and CouchDB obviously, but there's a big hurdle between
> the handful of erlang books, and real-world expert development.
>
> My (selfish!) original intent of asking for this list to be created was
> to use CouchDB as a convenient playground / reference point for improving
> my erlangz, and ultimately end up with some contributable code. And I
> hope that this approach will be a great thing for many of us in the
> same situation, and in the long term, also grow our community &
> committers too.
>
> So I am *really* keen for this to be as much of a group approach as possible,
> and spend time reviewing other's solutions etc, more so than just hacking
> away in a closet. We could set up a shared git repo if you all like, or
> just swap patches/gists over email. Whatever works.
>
> If you like the approach, the first one I suggest is about configuration, and we
> can check back in a week. I'd be rapt if this works out and others
> propose some topics along the way too.
>
> Homework :-)
>
> 1. how are configs loaded at startup in CouchDB?
> 2. how they're subsequently mutated/abused over the _config REST API?
> 3. how are these events managed elsewhere e.g. in couch httpd when IP changes?
> 4. compare with the sys.config approach used in OTP [1]
> 5. can you see a way to have the best of both worlds?
>
> I'll send my findings through early next week.
>
> A+
> Dave
>
> [1]: http://www.erlang.org/doc/design_principles/applications.html#id74029
>
>
Homework.

Using rcouch on a raspbian Raspberry Pi
rcouch for this exercise is roughly Couchdb 1.3

Using erlang view server code

lists:foreach(fun(X)-> Log(io_lib:format("~p~n",[ets:info(X)])) end, ets:all())

to list all ets tables, to find couch_config below


2012-11-18 21:23:03.904 [info] <0.2383.78>@couch_native_process:bindings:273
[{compressed,false},
  {memory,11004},
  {owner,<0.133.0>},
  {heir,none},
  {name,couch_config},
  {size,97},
  {node,'rcouch@127.0.0.1'},
  {named_table,true},                    %Name can be used to access
  {type,set},                                   %fast access
  {keypos,1},                                  %key is tuple of two strings (really lists)
  {protection,protected}]               %any process that has table identifier can read

then
ets:foldl(fun(Z,A)->Log(io_lib:format("~p~n",[Z])) end, [],couch_config)

to list all keys (which are tuples of strings) and values (tuples of various forms)

All output of is to couch.log (truncated for brevity)

2012-11-18 21:33:
39.835 [info] <0.2383.78>@couch_native_process:bindings:273

{{"httpd_db_handlers","_view_cleanup"},"{couch_mrview_http, handle_cleanup_req}"}
39.839 {{"daemons","compaction_daemon"},"{couch_compaction_daemon, start_link, []}"}
39.846 {{"couch_httpd_auth","iterations"},"10000"}
39.854 {{"attachments","compression_level"},"8"}
39.861 {{"replicator","http_connections"},"20"}
39.868 {{"httpd_db_handlers","_spatial_cleanup"},"{refuge_spatial_http, handle_cleanup_req}"}
39.875 {{"couchdb","delayed_commits"},"true"}
39.882 {{"httpd_db_handlers","_random_doc"},"{couch_randomdoc_httpd, handle_req}"}
39.889 {{"httpd_global_handlers","_config"},"{couch_httpd_misc_handlers, handle_config_req}"}
39.896 {{"log","level"},"info"}
39.903 {{"couchdb","uri_file"},"./data/couch.uri"}
39.910 {{"httpd_db_handlers","_all_docs"},"{couch_mrview_http, handle_all_docs_req}"}
39.917 {{"httpd","log_max_chunk_size"},"1000000"}
39.926 {{"couch_httpd_auth","authentication_redirect"},"/_utils/session.html"}
39.929 {{"couchdb","max_dbs_open"},"100"}
39.932 {{"httpd_global_handlers","_replicate"},"{couch_replicator_httpd, handle_req}"}
39.934 {{"vendor","version"},"0.6.0"}
39.938 {{"stats","samples"},"[0, 60, 300, 900]"}
39.945 {{"query_servers","javascript"},"./bin/couchjs ./share/server/main.js"}
39.952 {{"couchdb","database_dir"},"./data"}
39.959 {{"attachments","compressible_types"},"text/*, application/javascript, application/json,
application/xml"}
39.966 {{"uuids","algorithm"},"sequential"}
39.972 {{"couch_httpd_auth","authentication_db"},"_users"}
39.980 {{"httpd_global_handlers","_log"},"{couch_httpd_misc_handlers, handle_log_req}"}
39.987 {{"replicator","connection_timeout"},"60000"}
39.994 {{"httpd_design_handlers","_spatial"},"{refuge_spatial_http, handle_spatial_req}"}
40.001 {{"httpd","authentication_handlers"},
        "{couch_httpd_oauth, oauth_authentication_handler},
        {couch_httpd_auth, cookie_authentication_handler},
        {couch_httpd_auth, default_authentication_handler}"}
40.008 {{"compaction_daemon","min_file_size"},"131072"}
40.015 {{"httpd_design_handlers","_list"}, "{couch_mrview_show, handle_view_list_req}"}
40.022 {{"httpd_design_handlers","_spatial/_list"},"{refuge_spatial_show, handle_spatial_list_req}"}
40.029 {{"daemons","external_manager"},"{couch_external_manager, start_link, []}"}
40.036 {{"vendor","name"},"rcouch"}
40.043 {{"httpd","allow_jsonp"},"false"}
40.050 {{"httpd_global_handlers","_oauth"},"{couch_httpd_oauth, handle_oauth_req}"}
40.058 {{"httpd_global_handlers","_stats"},"{couch_httpd_stats_handlers, handle_stats_req}"}
40.065 {{"httpd_global_handlers","/"},"{couch_httpd_misc_handlers, handle_welcome_req, <<\"Welcome\">>}"}
40.072 {{"stats","rate"},"1000"}
40.079 {{"httpd_global_handlers","_uuids"},"{couch_httpd_misc_handlers, handle_uuids_req}"}
40.086 {{"httpd_global_handlers","_utils"},"{couch_httpd_misc_handlers, handle_utils_dir_req,
\"./share/www\"}"}
40.094 {{"httpd_db_handlers","_changes"},"{couch_httpd_changes, handle_changes_req}"}
40.101 {{"httpd_design_handlers","_view"},"{couch_mrview_http, handle_view_req}"}
40.108 {{"query_server_config","reduce_limit"},"true"}
40.115 {{"httpd_global_handlers","favicon.ico"},"{couch_httpd_misc_handlers, handle_favicon_req,
\"./share/www\"}"}
40.122 {{"log","file"},"./log/console.log"}
40.124 {{"couch_httpd_auth","require_valid_user"},"false"}
40.127 {{"httpd","secure_rewrites"},"true"}
40.134 {{"couch_httpd_auth","secret"},"c6e697df9db3a0b9ee7c62fc6657bc9c"}
40.140 {{"admins","david"},
"-pbkdf2-1a549048f941268c8c29140c4500a0a204dc6a88,3682c7bbbe8607b10088467218b7023c,10000"}
40.147 {{"httpd_global_handlers","_db_updates"},"{couch_dbupdates_httpd, handle_req}"}
40.154 {{"httpd_db_handlers","_compact"},"{couch_httpd_db, handle_compact_req}"}
40.162 {{"ssl","enable"},"false"}
40.169 {{"httpd_db_handlers","_temp_view"},"{couch_mrview_http, handle_temp_view_req}"}
40.181 {{"couch_httpd_oauth","use_users_db"},"false"}
40.184 {{"query_servers","coffeescript"},"./bin/couchjs ./share/server/main-coffee.js"}
40.187 {{"couch_httpd_auth","timeout"},"600"}
40.191 {{"replicator","db"},"_replicator"}
40.199 {{"httpd_global_handlers","_restart"},"{couch_httpd_misc_handlers, handle_restart_req}"}
40.206 {{"httpd_design_handlers","_compact"},"{couch_mrview_http, handle_compact_req}"}
40.213 {{"log","include_sasl"},"true"}
40.224 {{"ssl","ssl_certificate_max_depth"},"1"}
40.232 {{"httpd_design_handlers","_info"},"{couch_mrview_http, handle_info_req}"}
40.239 {{"database_compaction","doc_buffer_size"},"524288"}
40.248 {{"httpd","default_handler"},"{couch_httpd_db, handle_request}"}
40.255 {{"httpd_design_handlers","_spatial/_info"},"{refuge_spatial_http, handle_info_req}"}
40.264 {{"httpd","port"},"5984"}
40.269 {{"httpd_global_handlers","_active_tasks"}, "{couch_httpd_misc_handlers, handle_task_status_req}"}
40.279 {{"couchdb","max_document_size"},"4294967296"}
40.285 {{"daemons","query_servers"},"{couch_query_servers, start_link, []}"}
40.292 {{"httpd_db_handlers","_design"},"{couch_httpd_db, handle_design_req}"}
40.300 {{"couchdb","os_process_timeout"},"500000"}
40.308 {{"httpd_design_handlers","_rewrite"},"{couch_httpd_rewrite, handle_rewrite_req}"}
40.314 {{"replicator","socket_options"},"[{keepalive, true}, {nodelay, false}]"}
40.322 {{"httpd_design_handlers","_spatial/_compact"},"{refuge_spatial_http, handle_compact_req}"}
40.329 {{"httpd","vhost_global_handlers"},"_utils, _uuids, _session, _oauth, _users"}
40.336 {{"daemons","os_daemons"},"{couch_os_daemons, start_link, []}"}
40.343 {{"httpd_design_handlers","_update"},"{couch_mrview_show, handle_doc_update_req}"}
40.350 {{"httpd","bind_address"},"0.0.0.0"}
40.357 {{"ssl","verify_ssl_certificates"},"false"}
40.364 {{"httpd_design_handlers","_random"},"{couch_randomdoc_show, handle_randomdoc_show_req}"}
40.371 {{"replicator","max_replication_retry_count"},"10"}
40.377 {{"ssl","port"},"6986"}
40.385 {{"database_compaction","checkpoint_after"},"5242880"}
40.392 {{"replicator","verify_ssl_certificates"},"false"}
40.400 {{"view_compaction","keyvalue_buffer_size"},"2097152"}
40.402 {{"replicator","retries_per_request"},"10"}
40.405 {{"httpd_design_handlers","_show"},"{couch_mrview_show, handle_doc_show_req}"}
40.408 {{"couchdb","index_dir"},"./data"}
40.411 {{"couchdb","file_compression"},"snappy"}
40.413 {{"replicator","worker_processes"},"4"}
40.416 {{"native_query_servers","erlang"},"{couch_native_process, start_link, []}"}
40.425 {{"couch_httpd_auth","auth_cache_size"},"50"}
40.432 {{"httpd_global_handlers","_all_dbs"},"{couch_httpd_misc_handlers, handle_all_dbs_req}"}
40.463 {{"replicator","worker_batch_size"},"500"}
40.481 {{"compaction_daemon","check_interval"},"300"}
40.495 {{"replicator","ssl_certificate_max_depth"},"3"}
40.517 {{"httpd_global_handlers","_session"},"{couch_httpd_auth, handle_session_req}"}
        {{"query_server_config","os_process_limit"},"25"}

The reading of default.ini and local.ini into this ets table is another exercise, but it should
be noted
that as the couch_config table is of type "set" (all keys must be unique)
any entry from default.ini will be overwritten by any entry from
local.ini or any changes in Configuration from Futon etc.

Information in couch_config is available to all processes in couchdb without needing to be
passed any
data. Thus it is the Erlang way of having a shared fast look-up table in memory. Joe Armstrong
quotes
that a set lookup took 0.5 microsecond on his machine (2007) "Programming Erlang- Software
for a concurrent world"
pp. 277

I hope this helps someone else

David Martin (davidoccam)

Mime
View raw message