couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlos Alonso <carlos.alo...@cabify.com>
Subject Re: Understanding a couple replication errors
Date Mon, 31 Jul 2017 13:58:09 GMT
On top of those, this one happens from time to time as well:

[error] 2017-07-31T13:41:50.258423Z couchdb@couch-2 emulator -------- Error
in process <0.16620.2434> on node 'couchdb@couch-2' with exit value:
{{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},...

[error] 2017-07-31T13:41:50.260569Z couchdb@couch-2 <0.32053.2433> --------
Replicator, request PUT to "http://admin:*****@
127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false"
failed due to error {error,
    {'EXIT',
        {{{nocatch,{mp_parser_died,noproc}},
          [{couch_att,'-foldl/4-fun-0-',3,
               [{file,"src/couch_att.erl"},{line,591}]},
           {couch_att,fold_streamed_data,4,
               [{file,"src/couch_att.erl"},{line,642}]},
           {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
           {couch_httpd_multipart,atts_to_mp,4,
               [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
         {gen_server,call,
             [<0.11924.2435>,
              {send_req,
                  {{url,
                       "
http://admin:password@127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false
",
                       "127.0.0.1",5984,"admin","password",

 "/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false",
                       http,ipv4_address},
                   [{"Accept","application/json"},
                    {"Content-Length",32845},
                    {"Content-Type",
                     "multipart/related;
boundary=\"46b37c9ba185ac67bf24ee40bcc38828\""},
                    {"User-Agent","CouchDB-Replicator/2.0.0"}],
                   put,
                   {#Fun<couch_replicator_api_wrap.11.79455979>,

{<<"{\"_id\":\"cba0aa16e347ac7d6b09732032828704\",\"_rev\":\"8-b90e6bd0b6281f303e587f4b9538a0af\",...\"_revisions\":{\"start\":8,\"ids\":[\"b90e6bd0b6281f303e587f4b9538a0af\",\"df66a380ea1258243bfd8a33506a18b4\",\"4cd932d1b71cebc4a55f91331d97f902\",\"a4dabd1dbf2b1de8987119a65cb44131\",\"82cc46c88ca768ff092de95c338bc0e9\",\"b4a7562ac8a831f46a194e97ea33e527\",\"d4fd4ddb998e4022e0335f609ed1ef50\",\"118cae6fc6864b3c48f788e78705c8cb\"]},\"_attachments\":{\"FF01-0020941.pdf\":{\"content_type\":\"application/pdf\",\"revpos\":8,\"digest\":\"md5-IT8Ms2HDG9EFLvfIxYTaVw==\",\"length\":21888,\"follows\":true},\"FF01-0020941.xml\":{\"content_type\":\"application/xml\",\"revpos\":8,\"digest\":\"md5-KJIXHDQbcJ3xeLaOUh11cw==\",\"length\":8526,\"follows\":true}}}">>,
                     [{att,<<"FF01-0020941.pdf">>,<<"application/pdf">>,
                          21888,21888,
                          <<33,63,12,179,97,195,27,209,5,46,247,200,197,132,
                            218,87>>,
                          8,
                          {follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
                          identity},

{att,<<"FF01-0020941.xml">>,<<"application/xml">>,8526,
                          8526,

<<40,146,23,28,52,27,112,157,241,120,182,142,82,29,
                            117,115>>,
                          8,
                          {follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
                          identity}],
                     <<"46b37c9ba185ac67bf24ee40bcc38828">>,32845}},
                   [{response_format,binary},
                    {inactivity_timeout,30000},
                    {socket_options,[{keepalive,true},{nodelay,false}]}],
                   infinity}},
              infinity]}}}}


Thanks!



On Mon, Jul 31, 2017 at 11:52 AM Carlos Alonso <carlos.alonso@cabify.com>
wrote:

> Hi all!!
>
> I've been observing logs for a while and trying to understand why a few
> errors happen while (pull) replication is happening and also trying to
> understand the consequences.
>
> ** connection_closed*
>
> Log:
> ```
> [error]  <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"
> failed due to error connection_closed
> ```
>
> Cause: This one happens from time to time. It looks like the source is
> closing connection, but why?
> Consequences: None. I think replicator just retries the task.
>
> * *changes_reader_died*
>
> Log:
> ```
> [error] <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"
> failed due to error connection_closed
>
> [error] <0.13952.1946> -------- ChangesReader process died with reason:
> {http_request_failed,"GET","
> https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed
> <https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000%22,%7Berror,connection_closed>
> }}
>
> [error]  <0.13952.1946> -------- Replication
> `363ed31e35361d024d22e0a990ca1ae5+continuous` (`https://admin:*****@source.ip:443/my_db/`
> -> `http://admin:*****@127.0.0.1:5984/my_db/`
> <http://127.0.0.1:5984/my_db/>) failed: changes_reader_died
>
> [error]  <0.30404.6252> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"
> failed due to error {error,closing_on_request}
>
> [error] <0.352.0> -------- Error in replication
> `363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document
> `28d973af08931f79fa423b54a8004b31`): changes_reader_died
> Restarting replication in 5 seconds.
>
> [error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with
> reason: changes_reader_died
>   last msg:
> {'EXIT',<0.15968.1946>,{http_request_failed,"GET","https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}}
>      state:
> [{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb,"https://admin:
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb,"http://admin:
> *****@127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>},"https://admin:
> *****@source.ip:443/my_db/","http://admin:*****@127.0.0.1:5984/my_db/
> ",{httpdb,"https://admin:
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:
> *****@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri,
> 28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017
02:28:59
> GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri,
> 28 Jul 2017 09:46:48
> GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}]
>
> [error] <0.13952.1946> -------- CRASH REPORT Process  (<0.13952.1946>)
> with 2 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator,init,['Argument__1']}, ancestors:
> [couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [],
> links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...],
> dictionary:
> [{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...],
> trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> reductions: 26502159
>
> [error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child
> "363ed31e35361d024d22e0a990ca1ae5+continuous" started with
> {gen_server,start_link,undefined} at <0.13952.1946> exit with reason
> changes_reader_died in context child_terminated
>
> [error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with
> reason: changes_reader_died
>   last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
>      state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\
> ",...}}
>
> [error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with
> reason: changes_reader_died
>   last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
>      state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\
> ",...],...}}
>
> [error]  <0.16617.1946> -------- CRASH REPORT Process  (<0.16617.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.21799.6253>,<0.24330.6253>,<0.27365.6253>,<0.28980.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381757,...}}],
> trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> reductions: 6921530028
>
> [error]  <0.16155.1946> -------- CRASH REPORT Process  (<0.16155.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.12924.6253>,<0.21982.6253>,<0.27692.6253>,<0.30450.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381751,...}}],
> trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> reductions: 6914251593
>
> [error] <0.18469.1946> -------- gen_server <0.18469.1946> terminated with
> reason: changes_reader_died
>   last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
>      state: {state,<0.13952.1946>,<0.21935.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0
> }}
>
> [error] <0.10009.1946> -------- gen_server <0.10009.1946> terminated with
> reason: changes_reader_died
>   last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
>      state: {state,<0.13952.1946>,<0.22823.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\
> ",,...],...}}
>
> [error] <0.18469.1946> -------- CRASH REPORT Process  (<0.18469.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.18441.6253>,<0.25304.6253>,<0.30475.6253>,<0.32462.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381762,...}}],
> trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> reductions: 6918077761
>
> [error]  <0.10009.1946> -------- CRASH REPORT Process  (<0.10009.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.18053.6253>,<0.23938.6253>,<0.29957.6253>,<0.31079.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381758,...}}],
> trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> reductions: 6919159795
> ```
>
> Cause: This one looks similar to the one before. The trigger seems to be
> that the source closes the connection for some reason, but the logs are
> much more verbose this time.
> Consequences: None. I think replicator just retries the task.
>
> Furthermore, this last logs show the username and passwords plain text on
> the state key.
>
> Sorry for the wall of text.
>
> Regards
>
> --
> [image: Cabify - Your private Driver] <http://www.cabify.com/>
>
> *Carlos Alonso*
> Data Engineer
> Madrid, Spain
>
> carlos.alonso@cabify.com
>
> Prueba gratis con este código
> #CARLOSA6319 <https://cabify.com/i/carlosa6319>
> [image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
> <http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
> Linkedin] <https://www.linkedin.com/in/mrcalonso>
>
-- 
[image: Cabify - Your private Driver] <http://www.cabify.com/>

*Carlos Alonso*
Data Engineer
Madrid, Spain

carlos.alonso@cabify.com

Prueba gratis con este código
#CARLOSA6319 <https://cabify.com/i/carlosa6319>
[image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
<http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
Linkedin] <https://www.linkedin.com/in/mrcalonso>

-- 
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su 
destinatario, pudiendo contener información confidencial sometida a secreto 
profesional. No está permitida su reproducción o distribución sin la 
autorización expresa de Cabify. Si usted no es el destinatario final por 
favor elimínelo e infórmenos por esta vía. 

This message and any attached file are intended exclusively for the 
addressee, and it may be confidential. You are not allowed to copy or 
disclose it without Cabify's prior written authorization. If you are not 
the intended recipient please delete it from your system and notify us by 
e-mail.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message