Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3BC1BD3E4 for ; Tue, 30 Oct 2012 03:24:48 +0000 (UTC) Received: (qmail 35203 invoked by uid 500); 30 Oct 2012 03:24:43 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 35107 invoked by uid 500); 30 Oct 2012 03:24:42 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 35060 invoked by uid 99); 30 Oct 2012 03:24:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Oct 2012 03:24:40 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of decstery@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Oct 2012 03:24:34 +0000 Received: by mail-ie0-f176.google.com with SMTP id k11so8801998iea.35 for ; Mon, 29 Oct 2012 20:24:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=nxLygNiIm0uHlwPE9MVV7WL1jmhqVxkDwBiDGap68ZU=; b=Rz9b5kQ5zSH5Tt9D9EipqTTzpsKIBqI41RKagYIWjoy5WcXk3MF14Zjh75RsIQIihs aaaRKmBfJ7Y2ERh+3+hzKEeTl4tSeqnOx3Fi1yVPiIGEbOztdwF4X59eh2NJO6V2JCD5 lT3oJq/wWrx8ztOCBBj95cpCmztM+n910EqTuEB9uqBU6qcLCj8DC1+V0fkTsRDzkvmt fhDE5vCXJpWwXCXWJuIwrVhyeh3fg09zNY6AHt7PiPx6DATRUT5s1h9jcPQ95c6+hVZl qDEwWSEgP4dhyV66dPWDag4VcXiCbRGOBkWUS8Ip8b2Se1G8xwpHCle/0g00M6U4G4R8 rvQg== MIME-Version: 1.0 Received: by 10.42.157.202 with SMTP id e10mr27263112icx.41.1351567453741; Mon, 29 Oct 2012 20:24:13 -0700 (PDT) Received: by 10.50.189.199 with HTTP; Mon, 29 Oct 2012 20:24:13 -0700 (PDT) In-Reply-To: References: Date: Tue, 30 Oct 2012 11:24:13 +0800 Message-ID: Subject: Re: Tools for extracting data from hadoop logs From: Binglin Chang To: user@hadoop.apache.org Content-Type: multipart/mixed; boundary=90e6ba6138d0f861fe04cd3e51a9 X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba6138d0f861fe04cd3e51a9 Content-Type: multipart/alternative; boundary=90e6ba6138d0f861fa04cd3e51a7 --90e6ba6138d0f861fa04cd3e51a7 Content-Type: text/plain; charset=UTF-8 Hi, I think you want to analyze hadoop job logs in jobtracker history folder? These logs are in a centralized folder and don't need tools like flume or scribe to gather them. I used to write a simple python script to parse those log files, and generate csv/json reports, basically you can use it to get execution time, counter, status of job, taks, attempts, maybe you can modify it to meet you needs. Thanks, Binglin On Tue, Oct 30, 2012 at 9:48 AM, bharath vissapragada < bharathvissapragada1990@gmail.com> wrote: > Hi list, > > Are the any tools for parsing and extracting data from Hadoop's Job Logs? > I want to do stuff like .. > > 1. Getting run time of each map/reduce task > 2. Total map/reduce tasks ran on a particular node in that job and some > similar stuff > > Any suggestions? > > Thanks > --90e6ba6138d0f861fa04cd3e51a7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi,

I think you want to analyze hadoop job logs in jobtr= acker history folder? These logs are in a centralized folder and don't = need tools like flume or scribe to gather them.
I used to write a= simple python script to parse those log files, and generate csv/json repor= ts, basically you can use it to get execution time, counter, status of job,= taks, attempts, maybe you can modify it to meet you needs.

Thanks,
Binglin


On Tue, Oct 30, 2012 at 9:48 AM, bharath vissapragada <bharathvissapragada1990@gmail.com> wrote:
Hi list,

Are the any tool= s for parsing and extracting data from Hadoop's Job Logs? I want to do = stuff like ..=C2=A0

1. Getting run time of each map/reduce task
2= . Total map/reduce tasks ran on a particular node in that job =C2=A0and som= e similar stuff

Any suggestions?

T= hanks

--90e6ba6138d0f861fa04cd3e51a7-- --90e6ba6138d0f861fe04cd3e51a9 Content-Type: application/octet-stream; name="job_history.py" Content-Disposition: attachment; filename="job_history.py" Content-Transfer-Encoding: base64 X-Attachment-Id: f_h8wg837d0 aW1wb3J0IHN5cwppbXBvcnQgb3MKaW1wb3J0IHJlCmltcG9ydCB0aW1lCmltcG9ydCB0aHJlYWRp bmcKZnJvbSB1dGlsIGltcG9ydCAqCgoKRGF5UGF0aFJlZ2V4ID0gcmUuY29tcGlsZShyJygoMTl8 MjApXGRcZCkvPygwWzEtOV18MTB8MTF8MTIpLz8oWzBdWzEtOV18WzEyXVswLTldfDMwfDMxKScp CmRlZiBwYXJzZV9kYXlfZnJvbV9wYXRoKHBhdGgpOgogICAgbSA9IERheVBhdGhSZWdleC5zZWFy Y2gocGF0aCkKICAgIGlmIG06CiAgICAgICAgcmV0dXJuICclcy0lcy0lcycgJSAobS5ncm91cCgx KSwgbS5ncm91cCgzKSwgbS5ncm91cCg0KSkKICAgIHJldHVybiBOb25lCgpKb2JJREluRmlsZU5h bWVSZWdleCA9IHJlLmNvbXBpbGUocidfam9iX1xkK19cZCtfJykKZGVmIHBhcnNlX2pvYmlkX2Zy b21fZmlsZW5hbWUoZik6CiAgICBpZiBmLmVuZHN3aXRoKCcueG1sJykgb3IgZi5lbmRzd2l0aCgn LmNyYycpOgogICAgICAgIHJldHVybiBOb25lCiAgICBtID0gSm9iSURJbkZpbGVOYW1lUmVnZXgu c2VhcmNoKGYpCiAgICBpZiBtOgogICAgICAgIHJldHVybiBtLmdyb3VwKDApWzE6LTFdCiAgICBy ZXR1cm4gTm9uZQoKSm9iQnJpZWZJbmZvUHJvcGVydGllcyA9ICgnZGF0ZScsICdqb2JuYW1lJywg J2pvYmlkJywgJ3VzZXInLCAnam9iX3ByaW9yaXR5JywgJ2pvYl9zdGF0dXMnLCAndG90YWxfbWFw cycsICd0b3RhbF9yZWR1Y2VzJywgJ2ZhaWxlZF9tYXBzJywgJ2ZhaWxlZF9yZWR1Y2VzJywgJ3N1 Ym1pdF90aW1lJywgJ2xhdW5jaF90aW1lJywgJ2ZpbmlzaF90aW1lJywgJ2hkZnNfYnl0ZXNfcmVh ZCcsICdoZGZzX2J5dGVzX3dyaXR0ZW4nLCAnbWFwX291dHB1dF9ieXRlcycpCgpjbGFzcyBIaXN0 b3J5UmVwbyhSZWNvcmQpOgogICAgZGVmIF9faW5pdF9fKHNlbGYsIHBhdGgsIHNjYW49RmFsc2Up OgogICAgICAgIFJlY29yZC5fX2luaXRfXyhzZWxmLCBbInBhdGgiLCAiZGF5cyIsICJzdGF0dXMi LCAibGFzdF9zY2FuIl0pCiAgICAgICAgc2VsZi5wYXRoID0gcGF0aAogICAgICAgIHNlbGYuZGF5 cyA9IHt9CiAgICAgICAgc2VsZi5fc2Nhbl9sb2NrID0gdGhyZWFkaW5nLlJMb2NrKCkKICAgICAg ICBzZWxmLnN0YXR1cyA9ICJJRExFIgogICAgICAgIHNlbGYubGFzdF9zY2FuID0gMAogICAgICAg IGlmIHNjYW46CiAgICAgICAgICAgIHNlbGYuc2NhbigpCiAgICBkZWYgdXBkYXRlKHNlbGYpOgog ICAgICAgIHJldHVybiBzZWxmLnNjYW4oKQogICAgZGVmIHNjYW4oc2VsZik6CiAgICAgICAgaWYg c2VsZi5zdGF0dXMgPT0gIlNDQU5OSU5HIiBvciBzZWxmLmxhc3Rfc2NhbiArIDUgPiB0aW1lLnRp bWUoKToKICAgICAgICAgICAgcmV0dXJuIEZhbHNlCiAgICAgICAgd2l0aCBzZWxmLl9zY2FuX2xv Y2s6CiAgICAgICAgICAgIGlmIHNlbGYubGFzdF9zY2FuICsgNSA+IHRpbWUudGltZSgpOgogICAg ICAgICAgICAgICAgcmV0dXJuIEZhbHNlCiAgICAgICAgICAgIHNlbGYuc3RhdHVzID0gIlNDQU5O SU5HIgogICAgICAgICAgICBmb3Igcm9vdCwgZGlycywgZmlsZXMgaW4gb3Mud2FsayhzZWxmLnBh dGgpOgogICAgICAgICAgICAgICAgZGF0ZXN0ciA9IHBhcnNlX2RheV9mcm9tX3BhdGgocm9vdCkg b3IgJ2RlZmF1bHQnCiAgICAgICAgICAgICAgICBmb3IgZiBpbiBmaWxlczoKICAgICAgICAgICAg ICAgICAgICBqb2JpZCA9IHBhcnNlX2pvYmlkX2Zyb21fZmlsZW5hbWUoZikKICAgICAgICAgICAg ICAgICAgICBpZiBub3Qgam9iaWQ6CiAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlCiAg ICAgICAgICAgICAgICAgICAgZGF5ID0gc2VsZi5kYXlzLmdldChkYXRlc3RyKQogICAgICAgICAg ICAgICAgICAgIGlmIGRheSA9PSBOb25lOgogICAgICAgICAgICAgICAgICAgICAgICBkYXkgPSBE YXkoZGF0ZXN0cikKICAgICAgICAgICAgICAgICAgICAgICAgc2VsZi5kYXlzW2RhdGVzdHJdID0g ZGF5CiAgICAgICAgICAgICAgICAgICAgamggPSBkYXkuam9icy5nZXQoam9iaWQpCiAgICAgICAg ICAgICAgICAgICAgaWYgamggPT0gTm9uZToKICAgICAgICAgICAgICAgICAgICAgICAgamggPSBK b2JIaXN0b3J5KGpvYmlkLCBvcy5wYXRoLmpvaW4ocm9vdCwgZikpCiAgICAgICAgICAgICAgICAg ICAgICAgIGRheS5qb2JzW2pvYmlkXSA9IGpoCiAgICAgICAgICAgIHNlbGYuc3RhdHVzID0gIklE TEUiCiAgICAgICAgICAgIHNlbGYubGFzdF9zY2FuID0gdGltZS50aW1lKCkKICAgICAgICAgICAg cmV0dXJuIFRydWUKICAgIGRlZiBnZXRfYnJpZWZfaW5mb190YWJsZShzZWxmKToKICAgICAgICBk YXRhID0gW10KICAgICAgICBmb3IgZGF5IGluIHNvcnRlZChzZWxmLmRheXMuaXRlcmtleXMoKSk6 CiAgICAgICAgICAgIGRhdGEuZXh0ZW5kKHNlbGYuZGF5c1tkYXldLmdldF9icmllZl9pbmZvX3Rh YmxlKClbImRhdGEiXSkKICAgICAgICByZXR1cm4geyJjb2x1bW5zIjogSm9iQnJpZWZJbmZvUHJv cGVydGllcywgImRhdGEiOiBkYXRhfQogICAgZGVmIHF1ZXJ5X2RhdGVfcmFuZ2Uoc2VsZiwgc3Rh cnRfZGF0ZSwgZW5kX2RhdGUpOgogICAgICAgIGRhdGEgPSBbXQogICAgICAgIGZvciBrIGluIHNv cnRlZChzZWxmLmRheXMuaXRlcmtleXMoKSk6CiAgICAgICAgICAgIGlmIGsgPT0gJ2RlZmF1bHQn OgogICAgICAgICAgICAgICAgZGF5ID0gc2VsZi5kYXlzW2tdCiAgICAgICAgICAgICAgICBqb2Jp ZHMgPSBzb3J0ZWQoZGF5LmpvYnMuaXRlcmtleXMoKSkKICAgICAgICAgICAgICAgIGZvciBqb2Jp ZCBpbiBqb2JpZHM6CiAgICAgICAgICAgICAgICAgICAgZCA9IGRheS5qb2JzW2pvYmlkXS5nZXRf ZGF0ZSgpCiAgICAgICAgICAgICAgICAgICAgaWYgKChub3Qgc3RhcnRfZGF0ZSkgb3IgZCA+PSBz dGFydF9kYXRlKSBhbmQgKChub3QgZW5kX2RhdGUpIG9yIGQgPCBlbmRfZGF0ZSk6CiAgICAgICAg ICAgICAgICAgICAgICAgIGJpID0gZGF5LmpvYnNbam9iaWRdLmdldF9icmllZl9pbmZvKCkKICAg ICAgICAgICAgICAgICAgICAgICAgZGF0YS5hcHBlbmQoYmkpCiAgICAgICAgICAgIGVsc2U6CiAg ICAgICAgICAgICAgICBpZiBrID49IHN0YXJ0X2RhdGUgYW5kIGsgPCBlbmRfZGF0ZToKICAgICAg ICAgICAgICAgICAgICBkYXRhLmV4dGVuZChzZWxmLmRheXNba10uZ2V0X2JyaWVmX2luZm9fdGFi bGUoKVsnZGF0YSddKQogICAgICAgIHJldHVybiB7ImNvbHVtbnMiOiBKb2JCcmllZkluZm9Qcm9w ZXJ0aWVzLCAiZGF0YSI6IGRhdGF9CgpjbGFzcyBEYXkoUmVjb3JkKToKICAgIGRlZiBfX2luaXRf XyhzZWxmLCBkYXkpOgogICAgICAgIFJlY29yZC5fX2luaXRfXyhzZWxmLCBbImRheSIsICJqb2Jz Il0pCiAgICAgICAgc2VsZi5kYXkgPSBkYXkKICAgICAgICBzZWxmLmpvYnMgPSB7fQogICAgZGVm IGdldF9icmllZl9pbmZvX3RhYmxlKHNlbGYpOgogICAgICAgIGpvYmlkcyA9IHNvcnRlZChzZWxm LmpvYnMuaXRlcmtleXMoKSkKICAgICAgICBkYXRhID0gW3NlbGYuam9ic1tqaWRdLmdldF9icmll Zl9pbmZvKCkgZm9yIGppZCBpbiBqb2JpZHNdCiAgICAgICAgcmV0dXJuIHsiY29sdW1ucyI6IEpv YkJyaWVmSW5mb1Byb3BlcnRpZXMsICJkYXRhIjogZGF0YX0KCktleVZhbHVlUGFpclJlZ2V4ID0g cmUuY29tcGlsZShyJyhcdyspPSIoW14iXSooXFwiW14iXSopKikiJykKZGVmIHBhcnNlX2hpc3Rv cnlfcmVjb3JkKGxpbmUpOgogICAgcmV0ID0ge30KICAgIGZpcnN0X3NwYWNlID0gbGluZS5maW5k KCcgJykKICAgIHJldFsnX1RZUEVfJ10gPSBsaW5lWzpmaXJzdF9zcGFjZV0KICAgIGZvciBtIGlu IEtleVZhbHVlUGFpclJlZ2V4LmZpbmRpdGVyKGxpbmVbZmlyc3Rfc3BhY2UrMTpdKToKICAgICAg ICByZXRbbS5ncm91cCgxKV0gPSBtLmdyb3VwKDIpCiAgICByZXR1cm4gcmV0CgoKQ291bnRlckZp bHRlciA9ICgnSERGU19CWVRFU19SRUFEJywgJ0hERlNfQllURVNfV1JJVFRFTicsICdNQVBfT1VU UFVUX0JZVEVTJykKQ291bnRlclJlZ2V4ID0gcmUuY29tcGlsZShyJ1xbXCgoW1x3X10rKVwpXChb XlwpXStcKVwoKFxkKylcKVxdJykKZGVmIHBhcnNlX2NvdW50ZXJzKHMpOgogICAgcmV0ID0ge30K ICAgIGZvciBtIGluIENvdW50ZXJSZWdleC5maW5kaXRlcihzKToKICAgICAgICBpZiAobm90IENv dW50ZXJGaWx0ZXIpIG9yIChtLmdyb3VwKDEpIGluIENvdW50ZXJGaWx0ZXIpOgogICAgICAgICAg ICByZXRbbS5ncm91cCgxKS5sb3dlcigpXSA9IGludChtLmdyb3VwKDIpKQogICAgcmV0dXJuIHJl dAoKZGVmIHVwZGF0ZV9qb2Ioam9iLCByZWNvcmQpOgogICAgZm9yIGssdiBpbiByZWNvcmQuaXRl cml0ZW1zKCk6CiAgICAgICAgaWYgayBpbiAoJ0pPQklEJywgJ1VTRVInLCAnSk9CX1NUQVRVUycs ICdKT0JfUFJJT1JJVFknKToKICAgICAgICAgICAgam9iW2subG93ZXIoKV0gPSB2CiAgICAgICAg ZWxpZiBrID09ICdKT0JOQU1FJzoKICAgICAgICAgICAgam9iWydqb2JuYW1lJ10gPSB2LmRlY29k ZSgnc3RyaW5nLWVzY2FwZScpCiAgICAgICAgZWxpZiBrIGluICgnU1VCTUlUX1RJTUUnLCAKICAg ICAgICAgICAgICAgICAgICdMQVVOQ0hfVElNRScsIAogICAgICAgICAgICAgICAgICAgJ0ZJTklT SF9USU1FJyk6CiAgICAgICAgICAgIGpvYltrLmxvd2VyKCldID0gaW50KHYpLzEwMDAKICAgICAg ICBlbGlmIGsgaW4gKCdUT1RBTF9NQVBTJywgCiAgICAgICAgICAgICAgICAgICAnVE9UQUxfUkVE VUNFUycsIAogICAgICAgICAgICAgICAgICAgJ0ZJTklTSEVEX01BUFMnLCAKICAgICAgICAgICAg ICAgICAgICdGSU5JU0hFRF9SRURVQ0VTJywgCiAgICAgICAgICAgICAgICAgICAnRkFJTEVEX01B UFMnLCAKICAgICAgICAgICAgICAgICAgICdGQUlMRURfUkVEVUNFUycpOgogICAgICAgICAgICBq b2Jbay5sb3dlcigpXSA9IGludCh2KQogICAgICAgIGVsaWYgayA9PSAnQ09VTlRFUlMnOgogICAg ICAgICAgICBmb3IgY2ssY3YgaW4gcGFyc2VfY291bnRlcnModikuaXRlcml0ZW1zKCk6CiAgICAg ICAgICAgICAgICBqb2JbY2tdID0gY3YKICAgIGlmIGpvYi5nZXQoJ3N1Ym1pdF90aW1lJyk6CiAg ICAgICAgam9iWydkYXRlJ10gPSB0aW1lLnN0cmZ0aW1lKCclWS0lbS0lZCAlSDolTTolUycsIHRp bWUubG9jYWx0aW1lKGpvYlsnc3VibWl0X3RpbWUnXSkpCgpkZWYgdXBkYXRlX3Rhc2sodGFzaywg cmVjb3JkKToKICAgIHBhc3MKCmRlZiBzaW1wbGlmeShpZCk6CiAgICByZXR1cm4gJ18nLmpvaW4o aWQuc3BsaXQoJ18nKVszOl0pCgpkZWYgdXBkYXRlX2F0dGVtcHQoYXR0ZW1wdCwgcmVjb3JkKToK ICAgIGZvciBrLHYgaW4gcmVjb3JkLml0ZXJpdGVtcygpOgogICAgICAgIGlmIGsgPT0gJ1RBU0tf QVRURU1QVF9JRCc6CiAgICAgICAgICAgIGF0dGVtcHRbJ2lkJ10gPSB2CiAgICAgICAgICAgIGZk cyA9IHYuc3BsaXQoJ18nKQogICAgICAgICAgICBhdHRlbXB0Wyd0YXNrX2luZGV4J10gPSBpbnQo ZmRzWy0yXSkKICAgICAgICAgICAgYXR0ZW1wdFsnYXR0ZW1wdF9pbmRleCddID0gaW50KGZkc1st MV0pCiAgICAgICAgZWxpZiBrID09ICdUQVNLX1NUQVRVUyc6CiAgICAgICAgICAgIGF0dGVtcHRb J3N0YXR1cyddID0gdgogICAgICAgIGVsaWYgayBpbiAoJ1RBU0tfVFlQRScsICdIT1NUTkFNRScp OgogICAgICAgICAgICBhdHRlbXB0W2subG93ZXIoKV0gPSB2CiAgICAgICAgZWxpZiBrIGluICgn U1RBUlRfVElNRScsICdGSU5JU0hfVElNRScsICdTSFVGRkxFX0ZJTklTSEVEJywgJ1NPUlRfRklO SVNIRUQnKToKICAgICAgICAgICAgYXR0ZW1wdFtrLmxvd2VyKCldID0gaW50KHYpLzEwMDAKICAg ICAgICBlbGlmIGsgPT0gJ0NPVU5URVJTJzoKICAgICAgICAgICAgZm9yIGNrLGN2IGluIHBhcnNl X2NvdW50ZXJzKHYpLml0ZXJpdGVtcygpOgogICAgICAgICAgICAgICAgYXR0ZW1wdFtja10gPSBj dgoKZGVmIHBhcnNlX2hpc3RvcnkoZmluLCBzaGFsbG93PUZhbHNlKToKICAgIHJldCA9IHt9CiAg ICByZXRbJ3Rhc2tzJ10gPSB7fQogICAgcmV0WydhdHRlbXB0cyddID0ge30KICAgIGZvciBsaW5l IGluIGZpbjoKICAgICAgICByZWNvcmQgPSBwYXJzZV9oaXN0b3J5X3JlY29yZChsaW5lKQogICAg ICAgIHJ0eXBlID0gcmVjb3JkWydfVFlQRV8nXQogICAgICAgIGlmIHJ0eXBlID09ICdNRVRBJzoK ICAgICAgICAgICAgY29udGludWUKICAgICAgICBlbGlmIHJ0eXBlID09ICdKb2InOgogICAgICAg ICAgICB1cGRhdGVfam9iKHJldCwgcmVjb3JkKQogICAgICAgICAgICBpZiBzaGFsbG93IGFuZCBy ZXQuZ2V0KCdsYXVuY2hfdGltZScpOgogICAgICAgICAgICAgICAgcmV0dXJuIHJldAogICAgICAg IGVsaWYgcnR5cGUgPT0gIlRhc2siOgogICAgICAgICAgICB0YXNraWQgPSBzaW1wbGlmeShyZWNv cmRbJ1RBU0tJRCddKQogICAgICAgICAgICB0YXNrID0gcmV0Wyd0YXNrcyddLmdldCh0YXNraWQp CiAgICAgICAgICAgIGlmIG5vdCB0YXNrOgogICAgICAgICAgICAgICAgdGFzayA9IHt9CiAgICAg ICAgICAgICAgICByZXRbJ3Rhc2tzJ11bdGFza2lkXSA9IHRhc2sKICAgICAgICAgICAgdXBkYXRl X3Rhc2sodGFzaywgcmVjb3JkKQogICAgICAgIGVsaWYgcnR5cGUgaW4gKCJNYXBBdHRlbXB0Iiwg IlJlZHVjZUF0dGVtcHQiKToKICAgICAgICAgICAgYWlkID0gc2ltcGxpZnkocmVjb3JkWydUQVNL X0FUVEVNUFRfSUQnXSkKICAgICAgICAgICAgYXR0ZW1wdCA9IHJldFsnYXR0ZW1wdHMnXS5nZXQo YWlkKQogICAgICAgICAgICBpZiBub3QgYXR0ZW1wdDoKICAgICAgICAgICAgICAgIGF0dGVtcHQg PSB7fQogICAgICAgICAgICAgICAgcmV0WydhdHRlbXB0cyddW2FpZF0gPSBhdHRlbXB0CiAgICAg ICAgICAgIHVwZGF0ZV9hdHRlbXB0KGF0dGVtcHQsIHJlY29yZCkKICAgICAgICBlbHNlOgogICAg ICAgICAgICBwYXNzCiAgICByZXR1cm4gcmV0CgpjbGFzcyBKb2JIaXN0b3J5KFJlY29yZCk6CiAg ICBjYWNoZSA9IHt9CiAgICBjYWNoZV9tYXhfY3NpemUgPSAyMDQ4CiAgICBAc3RhdGljbWV0aG9k CiAgICBkZWYgdXBkYXRlX2NhY2hlKCk6CiAgICAgICAgYyA9IEpvYkhpc3RvcnkuY2FjaGUKICAg ICAgICBzaXplID0gSm9iSGlzdG9yeS5jYWNoZV9tYXhfY3NpemUKICAgICAgICBpZiBsZW4oYykg PD0gc2l6ZToKICAgICAgICAgICAgcmV0dXJuCiAgICAgICAgaHMgPSBbICh2LCBrKSBmb3Igayx2 IGluIGMuaXRlcml0ZW1zKCkgXQogICAgICAgIGhzLnNvcnQoKQogICAgICAgICMgZW1wdHkgMS80 IGNhY2hlCiAgICAgICAgZm9yIGkgaW4gcmFuZ2UobGVuKGMpLXNpemUvNCozKToKICAgICAgICAg ICAgamggPSBjLmdldChoc1tpXVsxXSkKICAgICAgICAgICAgaWYgamg6CiAgICAgICAgICAgICAg ICB3aXRoIGpoLl9sb2FkX2xvY2s6CiAgICAgICAgICAgICAgICAgICAgamguX2hpc3RvcnkgPSBO b25lCiAgICBkZWYgX19pbml0X18oc2VsZiwgam9iaWQsIHBhdGgpOgogICAgICAgIFJlY29yZC5f X2luaXRfXyhzZWxmLCBbImpvYmlkIiwgInBhdGgiLCAiaGlzdG9yeSJdKQogICAgICAgIHNlbGYu am9iaWQgPSBqb2JpZAogICAgICAgIHNlbGYucGF0aCA9IHBhdGgKICAgICAgICBzZWxmLl9oaXN0 b3J5ID0gTm9uZQogICAgICAgIHNlbGYuX2xvYWRfbG9jayA9IHRocmVhZGluZy5STG9jaygpCiAg ICAgICAgc2VsZi5fYnJpZWZfaW5mbyA9IE5vbmUKICAgICAgICBzZWxmLl9kYXRlID0gTm9uZQog ICAgICAgIHNlbGYuX25hbWUgPSBOb25lCiAgICBkZWYgZ2V0X2hpc3Rvcnkoc2VsZik6CiAgICAg ICAgcmV0ID0gc2VsZi5faGlzdG9yeQogICAgICAgIGlmIG5vdCByZXQ6CiAgICAgICAgICAgIHdp dGggc2VsZi5fbG9hZF9sb2NrOgogICAgICAgICAgICAgICAgaWYgbm90IHNlbGYuX2hpc3Rvcnk6 CiAgICAgICAgICAgICAgICAgICAgc2VsZi5sb2FkKCkKICAgICAgICAgICAgICAgICAgICByZXQg PSBzZWxmLl9oaXN0b3J5CiAgICAgICAgSm9iSGlzdG9yeS5jYWNoZVtzZWxmLmpvYmlkXSA9IHRp bWUudGltZSgpCiAgICAgICAgcmV0dXJuIHJldAogICAgZGVmIGxvYWQoc2VsZik6CiAgICAgICAg Sm9iSGlzdG9yeS51cGRhdGVfY2FjaGUoKQogICAgICAgIGZpbiA9IG9wZW4oc2VsZi5wYXRoKQog ICAgICAgIHNlbGYuX2hpc3RvcnkgPSBwYXJzZV9oaXN0b3J5KGZpbikKICAgICAgICBzZWxmLl9i cmllZl9pbmZvID0gW3NlbGYuX2hpc3RvcnkuZ2V0KGspIGZvciBrIGluIEpvYkJyaWVmSW5mb1By b3BlcnRpZXNdCiAgICAgICAgc2VsZi5fZGF0ZSA9IHNlbGYuX2hpc3RvcnlbJ2RhdGUnXQogICAg ICAgIHNlbGYuX25hbWUgPSBzZWxmLl9oaXN0b3J5Wydqb2JuYW1lJ10KICAgICAgICBmaW4uY2xv c2UoKQogICAgZGVmIGdldF9icmllZl9pbmZvKHNlbGYpOgogICAgICAgIGlmIG5vdCBzZWxmLl9i cmllZl9pbmZvOgogICAgICAgICAgICB3aXRoIHNlbGYuX2xvYWRfbG9jazoKICAgICAgICAgICAg ICAgIGlmIG5vdCBzZWxmLl9icmllZl9pbmZvOgogICAgICAgICAgICAgICAgICAgIHNlbGYubG9h ZCgpCiAgICAgICAgcmV0dXJuIHNlbGYuX2JyaWVmX2luZm8KICAgIGRlZiBzaGFsbG93X2xvYWQo c2VsZik6CiAgICAgICAgZmluID0gb3BlbihzZWxmLnBhdGgpCiAgICAgICAgcmV0ID0gcGFyc2Vf aGlzdG9yeShmaW4sIFRydWUpCiAgICAgICAgc2VsZi5fZGF0ZSA9IHJldFsnZGF0ZSddCiAgICAg ICAgc2VsZi5fbmFtZSA9IHJldFsnam9ibmFtZSddCiAgICAgICAgZmluLmNsb3NlKCkKICAgIGRl ZiBnZXRfZGF0ZShzZWxmKToKICAgICAgICBpZiBub3Qgc2VsZi5fZGF0ZToKICAgICAgICAgICAg c2VsZi5zaGFsbG93X2xvYWQoKQogICAgICAgIHJldHVybiBzZWxmLl9kYXRlCiAgICBkZWYgZ2V0 X25hbWUoc2VsZik6CiAgICAgICAgaWYgbm90IHNlbGYuX25hbWU6CiAgICAgICAgICAgIHNlbGYu c2hhbGxvd19sb2FkKCkKICAgICAgICByZXR1cm4gc2VsZi5fbmFtZQoKCmRlZiBnZW5fYnJpZWZf cmVwb3J0KHJlcG9wYXRoLCBvcGF0aCwgc3RhcnRfZGF0ZT1Ob25lLCBlbmRfZGF0ZT1Ob25lKToK ICAgIGhyID0gSGlzdG9yeVJlcG8ocmVwb3BhdGgsIFRydWUpCiAgICB0YWJsZSA9IGhyLnF1ZXJ5 X2RhdGVfcmFuZ2Uoc3RhcnRfZGF0ZSwgZW5kX2RhdGUpCiAgICBpZiBvcGF0aC5lbmRzd2l0aCgi LmNzdiIpOgogICAgICAgIG9mID0gb3BlbihvcGF0aCwgJ3cnKQogICAgICAgIGR1bXBfanNvbih0 YWJsZSwgb2YpCiAgICAgICAgb2YuY2xvc2UoKQogICAgZWxpZiBvcGF0aC5lbmRzd2l0aCgiLmNz diIpOgogICAgICAgIG9mID0gb3BlbihvcGF0aCwgJ3cnKQogICAgICAgIHRhYmxlX3RvX2Nzdih0 YWJsZSwgb2YpCiAgICAgICAgb2YuY2xvc2UoKQogICAgZWxzZToKICAgICAgICByYWlzZSBFeGNl cHRpb24oIk91dHB1dCBmb3JtYXQgbm90IHN1cHBvcnQgJXMiICUgb3BhdGgpCiAgICByZXR1cm4g dGFibGUKCgojIEFzIGEgZ2xvYmFsIGhpc3RvcnkgcmVwbyBtb2R1bGUKX2RlZmF1bHRfcmVwbyA9 IE5vbmUKCmRlZiBpbml0X3JlcG8ocGF0aCk6CiAgICBwcmludCAiU3RhcnQgdG8gaW5pdCByZXBv IgogICAgZ2xvYmFsIF9kZWZhdWx0X3JlcG8KICAgIF9kZWZhdWx0X3JlcG8gPSBIaXN0b3J5UmVw byhwYXRoLCBUcnVlKQogICAgcHJpbnQgIlJlcG8gaW5pdGlhbGl6ZWQiCiAgICAKZGVmIGdldF9y ZXBvKCk6CiAgICByZXR1cm4gX2RlZmF1bHRfcmVwbwoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9f IjoKICAgICMgZ2VuZXJhdGluZyBjc3YgcmVwb3J0cyBmb3IgYSBkYXRlIHJhbmdlCiAgICBpZiBs ZW4oc3lzLmFyZ3YpIDwgNToKICAgICAgICBwcmludCAiVXNhZ2U6ICVzIDxyZXBvIHBhdGg+IDxv dXRwdXQgcGF0aCAqLmpzb258Ki5jc3Y+IFtzdGFydF9kYXRlIFtlbmRfZGF0ZV1dIgogICAgICAg IHByaW50ICJFeGFtcGxlOiBweXRob24gJXMgLi9sb2dzL2hpc3Rvcnkgam9icy0yMDEyLTA4LTA4 LmNzdiAyMDEyLTA4LTA4IDIwMTItMDgwOSIKICAgICAgICBwcmludCAiICAgICAgICAgcHl0aG9u ICVzIC4vbG9ncy9oaXN0b3J5IGpvYnMtMjAxMi0wOC0wOC5qc29uIDIwMTItMDgtMDggTm9uZSIK ICAgICAgICBzeXMuZXhpdCgxKQogICAgc3RhcnRfZGF0ZSA9IHN5cy5hcmd2WzNdCiAgICBpZiBz dGFydF9kYXRlID09ICJOb25lIjoKICAgICAgICBzdGFydF9kYXRlID0gTm9uZQogICAgZW5kX2Rh dGUgPSBzeXMuYXJndls0XQogICAgaWYgZW5kX2RhdGUgPT0gIk5vbmUiOgogICAgICAgIGVuZF9k YXRlID0gTm9uZQogICAgdGFibGUgPSBnZW5fYnJpZWZfcmVwb3J0KHN5cy5hcmd2WzFdLCBzeXMu YXJndlsyXSwgc3RhcnRfZGF0ZSwgZW5kX2RhdGUpCiAgICBwcmludCAiRmluaXNoZWQsIHRvdGFs ICVkIGpvYnMiICUgbGVuKHRhYmxlWydkYXRhJ10pCgoKCg== --90e6ba6138d0f861fe04cd3e51a9 Content-Type: application/octet-stream; name="util.py" Content-Disposition: attachment; filename="util.py" Content-Transfer-Encoding: base64 X-Attachment-Id: f_h8wg9hk61 aW1wb3J0IGpzb24KaW1wb3J0IHRpbWUKCmNsYXNzIFJlY29yZChvYmplY3QpOgogICAgZGVmIF9f aW5pdF9fKHNlbGYsIGF0dHJzPU5vbmUpOgogICAgICAgIHNlbGYuX2F0dHJzID0gYXR0cnMgb3Ig W10KICAgIGRlZiBnZXRfYXR0cihzZWxmLCBuYW1lKToKICAgICAgICByZXQgPSBOb25lCiAgICAg ICAgaWYgaGFzYXR0cihzZWxmLCBuYW1lKToKICAgICAgICAgICAgcmV0ID0gZ2V0YXR0cihzZWxm LCBuYW1lKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIGdldHRlciA9ICJnZXRfIituYW1lCiAg ICAgICAgICAgIGlmIGhhc2F0dHIoc2VsZiwgZ2V0dGVyKToKICAgICAgICAgICAgICAgIHJldCA9 IGdldGF0dHIoc2VsZiwgZ2V0dGVyKQogICAgICAgIGlmIGNhbGxhYmxlKHJldCk6CiAgICAgICAg ICAgIHJldHVybiByZXQoKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIHJldHVybiByZXQKICAg IGRlZiB0b19kaWN0KHNlbGYpOgogICAgICAgIHJldCA9IHt9CiAgICAgICAgZm9yIHAgaW4gc2Vs Zi5fYXR0cnM6CiAgICAgICAgICAgIHJldFtwXSA9IHNlbGYuZ2V0X2F0dHIocCkKICAgICAgICBy ZXR1cm4gcmV0CgoKY2xhc3MgUmVjb3JkRW5jb2Rlcihqc29uLkpTT05FbmNvZGVyKToKICAgIGRl ZiBkZWZhdWx0KHNlbGYsIG8pOgogICAgICAgIGlmIGlzaW5zdGFuY2UobywgUmVjb3JkKToKICAg ICAgICAgICAgcmV0dXJuIG8udG9fZGljdCgpCiAgICAgICAgcmV0dXJuIGpzb24uSlNPTkVuY29k ZXIuZGVmYXVsdChzZWxmLCBvKQoKX3JlID0gUmVjb3JkRW5jb2RlcigKICAgIHNraXBrZXlzPUZh bHNlLAogICAgZW5zdXJlX2FzY2lpPVRydWUsCiAgICBjaGVja19jaXJjdWxhcj1UcnVlLAogICAg YWxsb3dfbmFuPVRydWUsCiAgICBpbmRlbnQ9Tm9uZSwKICAgIHNlcGFyYXRvcnM9Tm9uZSwKICAg IGVuY29kaW5nPSd1dGYtOCcsCiAgICBkZWZhdWx0PU5vbmUsCikKCmRlZiBkdW1wX2pzb24ob2Jq LCBmcCwgc2tpcGtleXM9RmFsc2UsIGVuc3VyZV9hc2NpaT1UcnVlLCBjaGVja19jaXJjdWxhcj1U cnVlLAogICAgICAgIGFsbG93X25hbj1UcnVlLCBjbHM9Tm9uZSwgaW5kZW50PU5vbmUsIHNlcGFy YXRvcnM9Tm9uZSwKICAgICAgICBlbmNvZGluZz0ndXRmLTgnLCBkZWZhdWx0PU5vbmUsICoqa3cp OgogICAgIyBjYWNoZWQgZW5jb2RlcgogICAgaWYgKG5vdCBza2lwa2V5cyBhbmQgZW5zdXJlX2Fz Y2lpIGFuZAogICAgICAgIGNoZWNrX2NpcmN1bGFyIGFuZCBhbGxvd19uYW4gYW5kCiAgICAgICAg Y2xzIGlzIE5vbmUgYW5kIGluZGVudCBpcyBOb25lIGFuZCBzZXBhcmF0b3JzIGlzIE5vbmUgYW5k CiAgICAgICAgZW5jb2RpbmcgPT0gJ3V0Zi04JyBhbmQgZGVmYXVsdCBpcyBOb25lIGFuZCBub3Qg a3cpOgogICAgICAgIGl0ZXJhYmxlID0gX3JlLml0ZXJlbmNvZGUob2JqKQogICAgZWxzZToKICAg ICAgICBpZiBjbHMgaXMgTm9uZToKICAgICAgICAgICAgY2xzID0gUmVjb3JkRW5jb2RlcgogICAg ICAgIGl0ZXJhYmxlID0gY2xzKHNraXBrZXlzPXNraXBrZXlzLCBlbnN1cmVfYXNjaWk9ZW5zdXJl X2FzY2lpLAogICAgICAgICAgICBjaGVja19jaXJjdWxhcj1jaGVja19jaXJjdWxhciwgYWxsb3df bmFuPWFsbG93X25hbiwgaW5kZW50PWluZGVudCwKICAgICAgICAgICAgc2VwYXJhdG9ycz1zZXBh cmF0b3JzLCBlbmNvZGluZz1lbmNvZGluZywKICAgICAgICAgICAgZGVmYXVsdD1kZWZhdWx0LCAq Kmt3KS5pdGVyZW5jb2RlKG9iaikKICAgICMgY291bGQgYWNjZWxlcmF0ZSB3aXRoIHdyaXRlbGlu ZXMgaW4gc29tZSB2ZXJzaW9ucyBvZiBQeXRob24sIGF0CiAgICAjIGEgZGVidWdnYWJpbGl0eSBj b3N0CiAgICBmb3IgY2h1bmsgaW4gaXRlcmFibGU6CiAgICAgICAgZnAud3JpdGUoY2h1bmspCgpk ZWYgZHVtcHNfanNvbihvYmosIHNraXBrZXlzPUZhbHNlLCBlbnN1cmVfYXNjaWk9VHJ1ZSwgY2hl Y2tfY2lyY3VsYXI9VHJ1ZSwKICAgICAgICBhbGxvd19uYW49VHJ1ZSwgY2xzPU5vbmUsIGluZGVu dD1Ob25lLCBzZXBhcmF0b3JzPU5vbmUsCiAgICAgICAgZW5jb2Rpbmc9J3V0Zi04JywgZGVmYXVs dD1Ob25lLCAqKmt3KToKICAgICMgY2FjaGVkIGVuY29kZXIKICAgIGlmIChub3Qgc2tpcGtleXMg YW5kIGVuc3VyZV9hc2NpaSBhbmQKICAgICAgICBjaGVja19jaXJjdWxhciBhbmQgYWxsb3dfbmFu IGFuZAogICAgICAgIGNscyBpcyBOb25lIGFuZCBpbmRlbnQgaXMgTm9uZSBhbmQgc2VwYXJhdG9y cyBpcyBOb25lIGFuZAogICAgICAgIGVuY29kaW5nID09ICd1dGYtOCcgYW5kIGRlZmF1bHQgaXMg Tm9uZSBhbmQgbm90IGt3KToKICAgICAgICByZXR1cm4gX3JlLmVuY29kZShvYmopCiAgICBpZiBj bHMgaXMgTm9uZToKICAgICAgICBjbHMgPSBSZWNvcmRFbmNvZGVyCiAgICByZXR1cm4gY2xzKAog ICAgICAgIHNraXBrZXlzPXNraXBrZXlzLCBlbnN1cmVfYXNjaWk9ZW5zdXJlX2FzY2lpLAogICAg ICAgIGNoZWNrX2NpcmN1bGFyPWNoZWNrX2NpcmN1bGFyLCBhbGxvd19uYW49YWxsb3dfbmFuLCBp bmRlbnQ9aW5kZW50LAogICAgICAgIHNlcGFyYXRvcnM9c2VwYXJhdG9ycywgZW5jb2Rpbmc9ZW5j b2RpbmcsIGRlZmF1bHQ9ZGVmYXVsdCwKICAgICAgICAqKmt3KS5lbmNvZGUob2JqKQoKCmRlZiB0 b19jc3ZfdmFsdWUodik6CiAgICBpZiB2ID09IE5vbmU6CiAgICAgICAgcmV0dXJuICcnCiAgICBp ZiBpc2luc3RhbmNlKHYsIHN0cik6CiAgICAgICAgcmV0dXJuIHYucmVwbGFjZSgnLCcsICcnKQog ICAgcmV0dXJuIHN0cih2KQoKZGVmIHRhYmxlX3RvX2Nzdih0YWJsZSwgZm91dCk6CiAgICBmID0g RmFsc2UKICAgIGlmIGlzaW5zdGFuY2UoZm91dCwgc3RyKToKICAgICAgICBmb3V0ID0gb3Blbihm b3V0LCAndycpCiAgICAgICAgZiA9IFRydWUKICAgIGZvdXQud3JpdGUoIiwiLmpvaW4odGFibGVb ImNvbHVtbnMiXSkpCiAgICBmb3V0LndyaXRlKCJcbiIpCiAgICBmb3IgZSBpbiB0YWJsZVsiZGF0 YSJdOgogICAgICAgIGZvdXQud3JpdGUoIiwiLmpvaW4oW3RvX2Nzdl92YWx1ZShjKSBmb3IgYyBp biBlXSkpCiAgICAgICAgZm91dC53cml0ZSgiXG4iKQogICAgaWYgZjoKICAgICAgICBmb3V0LmNs b3NlKCkKCmRlZiBsb2FkX2NzdihmaW4pOgogICAgZiA9IEZhbHNlCiAgICBpZiBpc2luc3RhbmNl KGZpbiwgc3RyKToKICAgICAgICBmaW4gPSBvcGVuKGZpbikKICAgICAgICBmID0gVHJ1ZQogICAg ZGVmIHBsaW5lKGxpbmUpOgogICAgICAgIGlmIGxpbmVbLTFdID09ICdcbic6CiAgICAgICAgICAg IGxpbmUgPSBsaW5lWzotMV0KICAgICAgICBmcyA9IGxpbmUuc3BsaXQoJywnKQogICAgICAgIHJl dHVybiBbaW50KGUpIGlmIGUuaXNkaWdpdCgpIGVsc2UgZSBmb3IgZSBpbiBmc10KICAgIGNvbHVt bnMgPSBwbGluZShmaW4ucmVhZGxpbmUoKSkKICAgIGRhdGEgPSBbXQogICAgZm9yIGxpbmUgaW4g ZmluOgogICAgICAgIGRhdGEuYXBwZW5kKHBsaW5lKGxpbmUpKQogICAgaWYgZjoKICAgICAgICBm aW4uY2xvc2UoKQogICAgcmV0dXJuIHsnY29sdW1ucyc6Y29sdW1ucywgJ2RhdGEnOmRhdGF9Cgpk ZWYgdGltZV90b19zdHJpbmcodCwgZnVsbD1UcnVlKToKICAgIHJldHVybiB0aW1lLnN0cmZ0aW1l KCIlbS0lZCAlSDolTTolUyIgaWYgZnVsbCBlbHNlICIlSDolTTolUyIsIHRpbWUubG9jYWx0aW1l KHQpKQoK --90e6ba6138d0f861fe04cd3e51a9--