Return-Path: Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: (qmail 28650 invoked from network); 2 Dec 2009 13:42:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 2 Dec 2009 13:42:01 -0000 Received: (qmail 48490 invoked by uid 500); 2 Dec 2009 13:42:01 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 48416 invoked by uid 500); 2 Dec 2009 13:42:00 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 48407 invoked by uid 99); 2 Dec 2009 13:41:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 13:41:59 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of guillaume.viland@orange-ftgroup.com designates 80.12.204.243 as permitted sender) Received: from [80.12.204.243] (HELO relais-inet.francetelecom.com) (80.12.204.243) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 13:41:50 +0000 Received: from omfeda06.si.francetelecom.fr (unknown [xx.xx.xx.199]) by omfeda13.si.francetelecom.fr (ESMTP service) with ESMTP id 64DB91904E5 for ; Wed, 2 Dec 2009 14:41:30 +0100 (CET) Received: from PUEXCH21.nanterre.francetelecom.fr (unknown [10.101.44.28]) by omfeda06.si.francetelecom.fr (ESMTP service) with ESMTP id 4747CC805F for ; Wed, 2 Dec 2009 14:41:30 +0100 (CET) Received: from PUEXCB1F.nanterre.francetelecom.fr ([10.101.44.9]) by PUEXCH21.nanterre.francetelecom.fr ([10.101.44.28]) with mapi; Wed, 2 Dec 2009 14:41:30 +0100 From: To: "mapreduce-user@hadoop.apache.org" Date: Wed, 2 Dec 2009 14:41:29 +0100 Subject: RE: How to write a custom input format and record reader to read multiple lines of text from files Thread-Topic: How to write a custom input format and record reader to read multiple lines of text from files Thread-Index: AcpyY2oMn4LV7Ek0Q8KBvyckwx48DAAAm/tAADsq9cA= Message-ID: <18746_1259761290_4B166E8A_18746_3477_1_8DC8587EB3873A4C912B5A9519750B812080B08214@PUEXCB1F.nanterre.francetelecom.fr> References: <1259657422.4141.19.camel@kunal-f2i3> <20589_1259659300_4B14E024_20589_7367_1_8DC8587EB3873A4C912B5A9519750B81207FA1CA80@PUEXCB1F.nanterre.francetelecom.fr> In-Reply-To: <20589_1259659300_4B14E024_20589_7367_1_8DC8587EB3873A4C912B5A9519750B81207FA1CA80@PUEXCB1F.nanterre.francetelecom.fr> Accept-Language: fr-FR Content-Language: fr-FR X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: fr-FR Content-Type: multipart/mixed; boundary="_003_8DC8587EB3873A4C912B5A9519750B812080B08214PUEXCB1Fnante_" MIME-Version: 1.0 X-PMX-Version: 5.5.7.378829, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2009.12.2.130316 X-Virus-Checked: Checked by ClamAV on apache.org --_003_8DC8587EB3873A4C912B5A9519750B812080B08214PUEXCB1Fnante_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sorry, in my previous post, i forgot the attachments. MultipleLineTextInputFormat allows to read N lines (customizable with the m= apred.textrecordreader.linecount property). The simple case when files are = non splittable (compressed or forced to non splittable as in MultipleLineTe= xtInputFormat) is implemented. The full implementation when files are splittable is much more difficult to= implement because of the need to manage the case where the record composed= of N lines overlap a split... I'm interested in the full implementation. -----Message d'origine----- De=A0: guillaume.viland@orange-ftgroup.com [mailto:guillaume.viland@orange-= ftgroup.com]=20 Envoy=E9=A0: mardi 1 d=E9cembre 2009 10:22 =C0=A0: mapreduce-user@hadoop.apache.org Objet=A0: RE: How to write a custom input format and record reader to read = multiple lines of text from files I've developed a version of a MultipleLineTextInputFormat for hadoop 0.19. = I think it is not perfect but it works for my needs. I've attached the code, feel free to improve or use it. Do not hesitate to = contact me if you improve the code. -----Message d'origine----- De=A0: Kunal Gupta [mailto:kunal@techlead-india.com]=20 Envoy=E9=A0: mardi 1 d=E9cembre 2009 09:50 =C0=A0: mapreduce-user@hadoop.apache.org Objet=A0: Re: How to write a custom input format and record reader to read = multiple lines of text from files NLineInputFormat will help in splitting N lines of text for each Mapper, but it will still pass single line of text to each call to the Map function. I want N lines of text to be passed as 'value' to the Map function. By extending FileInputFormat and RecordReader classes i am concatinating N lines of text and setting that as the 'value'. But this program is not running. Probably some initialization error. I am intimating the framework to use my extended classes as InputFormat: job.setInputFormatClass(MultiLineFileInputFormat.class); On Tue, 2009-12-01 at 13:53 +0530, Amogh Vasekar wrote: > Hi, > The NLineInputFormat (o.a.h.mapreduce.lib.input) achieves more or less > the same, and should help you guide writing custom input format :) >=20 > Amogh >=20 >=20 > On 12/1/09 11:47 AM, "Kunal Gupta" wrote: >=20 > Can someone explain how to override the "FileInputFormat" and > "RecordReader" in order to be able to read multiple lines of > text from > input files in a single map task? >=20=20=20=20=20=20=20=20=20 > Here the key will be the offset of the first line of text and > value will > be the N lines of text. >=20=20=20=20=20=20=20=20=20 > I have overridden the class FileInputFormat: >=20=20=20=20=20=20=20=20=20 > public class MultiLineFileInputFormat > extends FileInputFormat{ > ... > } >=20=20=20=20=20=20=20=20=20 > and implemented the abstract method: >=20=20=20=20=20=20=20=20=20 > public RecordReader createRecordReader(InputSplit split, > TaskAttemptContext context) > throws IOException, InterruptedException {...} >=20=20=20=20=20=20=20=20=20 > I have also overridden the recordreader class: >=20=20=20=20=20=20=20=20=20 > public class MultiLineFileRecordReader extends > RecordReader > {...} >=20=20=20=20=20=20=20=20=20 > and in the job configuration, specified this new InputFormat > class: >=20=20=20=20=20=20=20=20=20 > job.setInputFormatClass(MultiLineFileInputFormat.class); >=20=20=20=20=20=20=20=20=20 > -----------------------------------------------------------------= --------- > When I run this new map/reduce program, i get the following > java error: > -----------------------------------------------------------------= --------- > Exception in thread "main" java.lang.RuntimeException: > java.lang.NoSuchMethodException: CustomRecordReader > $MultiLineFileInputFormat.() > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtil= s.java:115) > at > org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:= 882) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.ja= va:779) > at > org.apache.hadoop.mapreduce.Job.submit(Job.java:432) > at > org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447) > at > CustomRecordReader.main(CustomRecordReader.java:257) > Caused by: java.lang.NoSuchMethodException: CustomRecordReader > $MultiLineFileInputFormat.() > at java.lang.Class.getConstructor0(Class.java:2706) > at > java.lang.Class.getDeclaredConstructor(Class.java:1985) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtil= s.java:109) > ... 5 more >=20=20=20=20=20=20=20=20=20 >=20=20=20=20=20=20=20=20=20 ********************************* This message and any attachments (the "message") are confidential and inten= ded solely for the addressees.=20 Any unauthorised use or dissemination is prohibited. Messages are susceptible to alteration.=20 France Telecom Group shall not be liable for the message if altered, change= d or falsified. If you are not the intended addressee of this message, please cancel it imm= ediately and inform the sender. ******************************** ********************************* This message and any attachments (the "message") are confidential and inten= ded solely for the addressees.=20 Any unauthorised use or dissemination is prohibited. Messages are susceptible to alteration.=20 France Telecom Group shall not be liable for the message if altered, change= d or falsified. If you are not the intended addressee of this message, please cancel it imm= ediately and inform the sender. ******************************** --_003_8DC8587EB3873A4C912B5A9519750B812080B08214PUEXCB1Fnante_ Content-Type: application/octet-stream; name="MultipleLineTextInputFormat.java" Content-Description: MultipleLineTextInputFormat.java Content-Disposition: attachment; filename="MultipleLineTextInputFormat.java"; size=3484; creation-date="Wed, 02 Dec 2009 14:39:05 GMT"; modification-date="Wed, 02 Dec 2009 11:48:04 GMT" Content-Transfer-Encoding: base64 cGFja2FnZSBjb20uZnQucG9ydGFsLmRvcC5tYXByZWQ7DQovKioNCiAqIENvcHlyaWdodCAoYykg MjAwOSwgIEZyYW5jZSBUZWxlY29tIC0gT3JhbmdlICh3d3cub3JhbmdlLmNvbSkgQWxsIHJpZ2h0 cyByZXNlcnZlZC4NCiAqDQogKiBSZWRpc3RyaWJ1dGlvbiBhbmQgdXNlIGluIHNvdXJjZSBhbmQg YmluYXJ5IGZvcm1zLCB3aXRoIG9yIHdpdGhvdXQgbW9kaWZpY2F0aW9uLA0KICogYXJlIHBlcm1p dHRlZCBwcm92aWRlZCB0aGF0IHRoZSBmb2xsb3dpbmcgY29uZGl0aW9ucyBhcmUgbWV0Og0KICog ICAtIFJlZGlzdHJpYnV0aW9ucyBvZiBzb3VyY2UgY29kZSBtdXN0IHJldGFpbiB0aGUgYWJvdmUg Y29weXJpZ2h0DQogKiAgICAgbm90aWNlLCB0aGlzIGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhl IGZvbGxvd2luZyBkaXNjbGFpbWVyLg0KICogICAtIFJlZGlzdHJpYnV0aW9ucyBpbiBiaW5hcnkg Zm9ybSBtdXN0IHJlcHJvZHVjZSB0aGUgYWJvdmUgY29weXJpZ2h0DQogKiAgICAgbm90aWNlLCB0 aGlzIGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhlIGZvbGxvd2luZyBkaXNjbGFpbWVyIGluIHRo ZQ0KICogICAgIGRvY3VtZW50YXRpb24gYW5kL29yIG90aGVyIG1hdGVyaWFscyBwcm92aWRlZCB3 aXRoIHRoZSBkaXN0cmlidXRpb24uDQogKiAgIC0gTmVpdGhlciB0aGUgbmFtZSBvZiBGcmFuY2Ug VGVsZWNvbSAtIE9yYW5nZSBub3IgdGhlDQogKiAgICAgbmFtZXMgb2YgaXRzIGNvbnRyaWJ1dG9y cyBtYXkgYmUgdXNlZCB0byBlbmRvcnNlIG9yIHByb21vdGUgcHJvZHVjdHMNCiAqICAgICBkZXJp dmVkIGZyb20gdGhpcyBzb2Z0d2FyZSB3aXRob3V0IHNwZWNpZmljIHByaW9yIHdyaXR0ZW4gcGVy bWlzc2lvbi4NCiAqDQogKiBUSElTIFNPRlRXQVJFIElTIFBST1ZJREVEIEJZIEZyYW5jZSBUZWxl Y29tIC0gT3JhbmdlICcnQVMgSVMnJyBBTkQgQU5ZDQogKiBFWFBSRVNTIE9SIElNUExJRUQgV0FS UkFOVElFUywgSU5DTFVESU5HLCBCVVQgTk9UIExJTUlURUQgVE8sIFRIRSBJTVBMSUVEDQogKiBX QVJSQU5USUVTIE9GIE1FUkNIQU5UQUJJTElUWSBBTkQgRklUTkVTUyBGT1IgQSBQQVJUSUNVTEFS IFBVUlBPU0UgQVJFDQogKiBESVNDTEFJTUVELiBJTiBOTyBFVkVOVCBTSEFMTCBGcmFuY2UgVGVs ZWNvbSAtIE9yYW5nZSBCRSBMSUFCTEUgRk9SIEFOWQ0KICogRElSRUNULCBJTkRJUkVDVCwgSU5D SURFTlRBTCwgU1BFQ0lBTCwgRVhFTVBMQVJZLCBPUiBDT05TRVFVRU5USUFMIERBTUFHRVMNCiAq IChJTkNMVURJTkcsIEJVVCBOT1QgTElNSVRFRCBUTywgUFJPQ1VSRU1FTlQgT0YgU1VCU1RJVFVU RSBHT09EUyBPUiBTRVJWSUNFUzsNCiAqIExPU1MgT0YgVVNFLCBEQVRBLCBPUiBQUk9GSVRTOyBP UiBCVVNJTkVTUyBJTlRFUlJVUFRJT04pIEhPV0VWRVIgQ0FVU0VEIEFORA0KICogT04gQU5ZIFRI RU9SWSBPRiBMSUFCSUxJVFksIFdIRVRIRVIgSU4gQ09OVFJBQ1QsIFNUUklDVCBMSUFCSUxJVFks IE9SIFRPUlQNCiAqIChJTkNMVURJTkcgTkVHTElHRU5DRSBPUiBPVEhFUldJU0UpIEFSSVNJTkcg SU4gQU5ZIFdBWSBPVVQgT0YgVEhFIFVTRSBPRiBUSElTDQogKiBTT0ZUV0FSRSwgRVZFTiBJRiBB RFZJU0VEIE9GIFRIRSBQT1NTSUJJTElUWSBPRiBTVUNIIERBTUFHRS4NCiovDQoNCmltcG9ydCBq YXZhLmlvLklPRXhjZXB0aW9uOw0KDQppbXBvcnQgb3JnLmFwYWNoZS5oYWRvb3AuZnMuRmlsZVN5 c3RlbTsNCmltcG9ydCBvcmcuYXBhY2hlLmhhZG9vcC5mcy5QYXRoOw0KaW1wb3J0IG9yZy5hcGFj aGUuaGFkb29wLmlvLkxvbmdXcml0YWJsZTsNCmltcG9ydCBvcmcuYXBhY2hlLmhhZG9vcC5pby5U ZXh0Ow0KaW1wb3J0IG9yZy5hcGFjaGUuaGFkb29wLmlvLmNvbXByZXNzLkNvbXByZXNzaW9uQ29k ZWNGYWN0b3J5Ow0KaW1wb3J0IG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5GaWxlSW5wdXRGb3Jt YXQ7DQppbXBvcnQgb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLkZpbGVTcGxpdDsNCmltcG9ydCBv cmcuYXBhY2hlLmhhZG9vcC5tYXByZWQuSW5wdXRTcGxpdDsNCmltcG9ydCBvcmcuYXBhY2hlLmhh ZG9vcC5tYXByZWQuSm9iQ29uZjsNCmltcG9ydCBvcmcuYXBhY2hlLmhhZG9vcC5tYXByZWQuSm9i Q29uZmlndXJhYmxlOw0KaW1wb3J0IG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5SZWNvcmRSZWFk ZXI7DQppbXBvcnQgb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlJlcG9ydGVyOw0KDQoNCi8qKg0K ICogVGV4dEZpbGVJbnB1dEZvcm1hdCB0byBwcm9jZXNzIGtleXMgYXMgb2Zmc2V0IGFuZCB2YWx1 ZSBhcyBjb25jYXRlbmF0aW9uIG9mIE4NCiAqIGxpbmVzLiA8Yj4gLy9XQVJOSU5HIHRoaXMgcmVh ZGVyIG9ubHkgd29ya3MgZm9yIG5vbiBzcGxpdHRhYmxlIGZpbGVzIChnemlwZWQpDQogKiA8Yj4g VE9ETyBhZGFwdCB0aGlzIGNsYXNzIHdoZW4gZmlsZXMgYXJlIHNwbGl0dGFibGUNCiAqIEBhdXRo b3IgZ3VpbGxhdW1lIHZpbGFuZA0KICogQGNvcHlyaWdodCBDb3B5cmlnaHQgKGMpIDIwMDksICBG cmFuY2UgVGVsZWNvbSAtIE9yYW5nZSAod3d3Lm9yYW5nZS5jb20pIGFsbCByaWdodHMgcmVzZXJ2 ZWQuDQogKi8NCg0KcHVibGljIGNsYXNzIE11bHRpcGxlTGluZVRleHRJbnB1dEZvcm1hdCBleHRl bmRzDQogICAgICAgIEZpbGVJbnB1dEZvcm1hdDxMb25nV3JpdGFibGUsIFRleHQ+IGltcGxlbWVu dHMgSm9iQ29uZmlndXJhYmxlIHsNCg0KICAgIC8vIG5vdCByZWFkIGJlY2F1c2UgaXNTcGxpdGFi bGUgaXMgZm9yY2VkIHRvIHJldHVybiBmYWxzZQ0KICAgIHByaXZhdGUgQ29tcHJlc3Npb25Db2Rl Y0ZhY3RvcnkgY29tcHJlc3Npb25Db2RlY3MgPSBudWxsOw0KDQogICAgcHVibGljIHZvaWQgY29u ZmlndXJlKEpvYkNvbmYgY29uZikgew0KICAgICAgICBjb21wcmVzc2lvbkNvZGVjcyA9IG5ldyBD b21wcmVzc2lvbkNvZGVjRmFjdG9yeShjb25mKTsNCiAgICB9DQoNCiAgICAvKioNCiAgICAgKiBG b3JjZSBmaWxlcyB0byBiZSB1bnNwbGl0dGFibGUgaW4gb3JkZXIgdG8gYXZvaWQgdG8gc3BsaXQg ZmlsZXMNCiAgICAgKiBpbiB0aGUgbWlkZGxlIG9mIHRoZSBOIGxpbmVzLiANCiAgICAgKi8NCiAg ICBwcm90ZWN0ZWQgYm9vbGVhbiBpc1NwbGl0YWJsZShGaWxlU3lzdGVtIGZzLCBQYXRoIGZpbGUp IHsNCiAgICAgICAgcmV0dXJuIGZhbHNlOw0KICAgIH0NCg0KICAgIHB1YmxpYyBSZWNvcmRSZWFk ZXI8TG9uZ1dyaXRhYmxlLCBUZXh0PiBnZXRSZWNvcmRSZWFkZXIoDQogICAgICAgICAgICBJbnB1 dFNwbGl0IGdlbmVyaWNTcGxpdCwgSm9iQ29uZiBqb2IsIFJlcG9ydGVyIHJlcG9ydGVyKQ0KICAg ICAgICAgICAgdGhyb3dzIElPRXhjZXB0aW9uIHsNCiAgICAgICAgcmVwb3J0ZXIuc2V0U3RhdHVz KGdlbmVyaWNTcGxpdC50b1N0cmluZygpKTsNCiAgICAgICAgcmV0dXJuIG5ldyBNdWx0aXBsZUxp bmVUZXh0UmVjb3JkUmVhZGVyKGpvYiwgKEZpbGVTcGxpdCkgZ2VuZXJpY1NwbGl0KTsNCiAgICB9 DQoNCn0NCg== --_003_8DC8587EB3873A4C912B5A9519750B812080B08214PUEXCB1Fnante_ Content-Type: application/octet-stream; name="MultipleLineTextRecordReader.java" Content-Description: MultipleLineTextRecordReader.java Content-Disposition: attachment; filename="MultipleLineTextRecordReader.java"; size=8125; creation-date="Wed, 02 Dec 2009 14:39:55 GMT"; modification-date="Wed, 02 Dec 2009 14:39:35 GMT" Content-Transfer-Encoding: base64 cGFja2FnZSBjb20uZnQucG9ydGFsLmRvcC5tYXByZWQ7DQovKioNCiAqIENvcHlyaWdodCAoYykg MjAwOSwgIEZyYW5jZSBUZWxlY29tIC0gT3JhbmdlICh3d3cub3JhbmdlLmNvbSkgQWxsIHJpZ2h0 cyByZXNlcnZlZC4NCiAqDQogKiBSZWRpc3RyaWJ1dGlvbiBhbmQgdXNlIGluIHNvdXJjZSBhbmQg YmluYXJ5IGZvcm1zLCB3aXRoIG9yIHdpdGhvdXQgbW9kaWZpY2F0aW9uLA0KICogYXJlIHBlcm1p dHRlZCBwcm92aWRlZCB0aGF0IHRoZSBmb2xsb3dpbmcgY29uZGl0aW9ucyBhcmUgbWV0Og0KICog ICAtIFJlZGlzdHJpYnV0aW9ucyBvZiBzb3VyY2UgY29kZSBtdXN0IHJldGFpbiB0aGUgYWJvdmUg Y29weXJpZ2h0DQogKiAgICAgbm90aWNlLCB0aGlzIGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhl IGZvbGxvd2luZyBkaXNjbGFpbWVyLg0KICogICAtIFJlZGlzdHJpYnV0aW9ucyBpbiBiaW5hcnkg Zm9ybSBtdXN0IHJlcHJvZHVjZSB0aGUgYWJvdmUgY29weXJpZ2h0DQogKiAgICAgbm90aWNlLCB0 aGlzIGxpc3Qgb2YgY29uZGl0aW9ucyBhbmQgdGhlIGZvbGxvd2luZyBkaXNjbGFpbWVyIGluIHRo ZQ0KICogICAgIGRvY3VtZW50YXRpb24gYW5kL29yIG90aGVyIG1hdGVyaWFscyBwcm92aWRlZCB3 aXRoIHRoZSBkaXN0cmlidXRpb24uDQogKiAgIC0gTmVpdGhlciB0aGUgbmFtZSBvZiBGcmFuY2Ug VGVsZWNvbSAtIE9yYW5nZSBub3IgdGhlDQogKiAgICAgbmFtZXMgb2YgaXRzIGNvbnRyaWJ1dG9y cyBtYXkgYmUgdXNlZCB0byBlbmRvcnNlIG9yIHByb21vdGUgcHJvZHVjdHMNCiAqICAgICBkZXJp dmVkIGZyb20gdGhpcyBzb2Z0d2FyZSB3aXRob3V0IHNwZWNpZmljIHByaW9yIHdyaXR0ZW4gcGVy bWlzc2lvbi4NCiAqDQogKiBUSElTIFNPRlRXQVJFIElTIFBST1ZJREVEIEJZIEZyYW5jZSBUZWxl Y29tIC0gT3JhbmdlICcnQVMgSVMnJyBBTkQgQU5ZDQogKiBFWFBSRVNTIE9SIElNUExJRUQgV0FS UkFOVElFUywgSU5DTFVESU5HLCBCVVQgTk9UIExJTUlURUQgVE8sIFRIRSBJTVBMSUVEDQogKiBX QVJSQU5USUVTIE9GIE1FUkNIQU5UQUJJTElUWSBBTkQgRklUTkVTUyBGT1IgQSBQQVJUSUNVTEFS IFBVUlBPU0UgQVJFDQogKiBESVNDTEFJTUVELiBJTiBOTyBFVkVOVCBTSEFMTCBGcmFuY2UgVGVs ZWNvbSAtIE9yYW5nZSBCRSBMSUFCTEUgRk9SIEFOWQ0KICogRElSRUNULCBJTkRJUkVDVCwgSU5D SURFTlRBTCwgU1BFQ0lBTCwgRVhFTVBMQVJZLCBPUiBDT05TRVFVRU5USUFMIERBTUFHRVMNCiAq IChJTkNMVURJTkcsIEJVVCBOT1QgTElNSVRFRCBUTywgUFJPQ1VSRU1FTlQgT0YgU1VCU1RJVFVU RSBHT09EUyBPUiBTRVJWSUNFUzsNCiAqIExPU1MgT0YgVVNFLCBEQVRBLCBPUiBQUk9GSVRTOyBP UiBCVVNJTkVTUyBJTlRFUlJVUFRJT04pIEhPV0VWRVIgQ0FVU0VEIEFORA0KICogT04gQU5ZIFRI RU9SWSBPRiBMSUFCSUxJVFksIFdIRVRIRVIgSU4gQ09OVFJBQ1QsIFNUUklDVCBMSUFCSUxJVFks IE9SIFRPUlQNCiAqIChJTkNMVURJTkcgTkVHTElHRU5DRSBPUiBPVEhFUldJU0UpIEFSSVNJTkcg SU4gQU5ZIFdBWSBPVVQgT0YgVEhFIFVTRSBPRiBUSElTDQogKiBTT0ZUV0FSRSwgRVZFTiBJRiBB RFZJU0VEIE9GIFRIRSBQT1NTSUJJTElUWSBPRiBTVUNIIERBTUFHRS4NCiovDQoNCmltcG9ydCBq YXZhLmlvLklPRXhjZXB0aW9uOw0KaW1wb3J0IGphdmEuaW8uSW5wdXRTdHJlYW07DQoNCmltcG9y dCBvcmcuYXBhY2hlLmNvbW1vbnMubG9nZ2luZy5Mb2c7DQppbXBvcnQgb3JnLmFwYWNoZS5jb21t b25zLmxvZ2dpbmcuTG9nRmFjdG9yeTsNCmltcG9ydCBvcmcuYXBhY2hlLmhhZG9vcC5jb25mLkNv bmZpZ3VyYXRpb247DQppbXBvcnQgb3JnLmFwYWNoZS5oYWRvb3AuZnMuRlNEYXRhSW5wdXRTdHJl YW07DQppbXBvcnQgb3JnLmFwYWNoZS5oYWRvb3AuZnMuRmlsZVN5c3RlbTsNCmltcG9ydCBvcmcu YXBhY2hlLmhhZG9vcC5mcy5QYXRoOw0KaW1wb3J0IG9yZy5hcGFjaGUuaGFkb29wLmlvLkxvbmdX cml0YWJsZTsNCmltcG9ydCBvcmcuYXBhY2hlLmhhZG9vcC5pby5UZXh0Ow0KaW1wb3J0IG9yZy5h cGFjaGUuaGFkb29wLmlvLmNvbXByZXNzLkNvbXByZXNzaW9uQ29kZWM7DQppbXBvcnQgb3JnLmFw YWNoZS5oYWRvb3AuaW8uY29tcHJlc3MuQ29tcHJlc3Npb25Db2RlY0ZhY3Rvcnk7DQppbXBvcnQg b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLkZpbGVTcGxpdDsNCmltcG9ydCBvcmcuYXBhY2hlLmhh ZG9vcC5tYXByZWQuUmVjb3JkUmVhZGVyOw0KDQovKioNCiAqIFRyZWF0cyBrZXlzIGFzIG9mZnNl dCBpbiBmaWxlIGFuZCB2YWx1ZSBhcyBjb25jYXRlbmF0aW9uIG9mIE4gbGluZXMuDQogKiBSZWFk IE4gbGluZXMgYXMgYSB2YWx1ZS4gVGhlIE4gdmFsdWUgaXMgZGVmYXVsdGVkIHRvIDEwLiBUaGlz IG1pZ2h0IGJlIGNvbmZpZ3VyZWQNCiAqIHVzaW5nICBtYXByZWQudGV4dHJlY29yZHJlYWRlci5s aW5lY291bnQgcHJvcGVydHkuDQogKiA8Yj4gLy9XQVJOSU5HIHRoaXMgcmVhZGVyIG9ubHkgd29y a3MgZm9yIG5vbiBzcGxpdHRhYmxlIGZpbGVzIChmb3IgaW5zdGFjZSBnemlwZWQpIDxiPg0KICog VE9ETyBhZGFwdCB0aGlzIGNsYXNzIHdoZW4gZmlsZXMgYXJlIHNwbGl0YWJsZSAgDQogKiBUaGlz IGNsYXNzIGlzIGluc3BpcmVkIGZyb20gTGluZVJlY29yZFJlYWRlci4NCiAqIEBhdXRob3IgZ3Vp bGxhdW1lIHZpbGFuZA0KICogQGNvcHlyaWdodCBDb3B5cmlnaHQgKGMpIDIwMDksICBGcmFuY2Ug VGVsZWNvbSAtIE9yYW5nZSAod3d3Lm9yYW5nZS5jb20pIGFsbCByaWdodHMgcmVzZXJ2ZWQuDQog Kg0KICovDQpwdWJsaWMgY2xhc3MgTXVsdGlwbGVMaW5lVGV4dFJlY29yZFJlYWRlciBpbXBsZW1l bnRzIFJlY29yZFJlYWRlcjxMb25nV3JpdGFibGUsIFRleHQ+IHsNCiAgICBwcml2YXRlIHN0YXRp YyBmaW5hbCBpbnQgREVGQVVMVF9MSU5FX05VTUJFUiA9IDEwOw0KDQogICAgcHJpdmF0ZSBzdGF0 aWMgZmluYWwgTG9nIExPRyA9IExvZ0ZhY3RvcnkuZ2V0TG9nKE11bHRpcGxlTGluZVRleHRSZWNv cmRSZWFkZXIuY2xhc3MNCiAgICAgICAgICAgIC5nZXROYW1lKCkpOw0KDQogICAgcHJpdmF0ZSBD b21wcmVzc2lvbkNvZGVjRmFjdG9yeSBjb21wcmVzc2lvbkNvZGVjcyA9IG51bGw7DQogICAgcHJp dmF0ZSBsb25nIHN0YXJ0Ow0KICAgIHByaXZhdGUgbG9uZyBwb3M7DQogICAgcHJpdmF0ZSBsb25n IGVuZDsNCiAgICBwcml2YXRlIExpbmVSZWFkZXIgaW47DQogICAgaW50IG1heExpbmVMZW5ndGg7 DQogICAgcHJpdmF0ZSBUZXh0IGxpbmVSZWFkID0gbmV3IFRleHQoKTsNCiAgICBwcml2YXRlIGlu dCBsaW5lc051bWJlcjsNCg0KDQoNCiAgICAvKioNCiAgICAgKiBBIGNsYXNzIHRoYXQgcHJvdmlk ZXMgYSBsaW5lIHJlYWRlciBmcm9tIGFuIGlucHV0IHN0cmVhbS4NCiAgICAgKiANCiAgICAgKiBA ZGVwcmVjYXRlZCBVc2Uge0BsaW5rIG9yZy5hcGFjaGUuaGFkb29wLnV0aWwuTGluZVJlYWRlcn0g aW5zdGVhZC4NCiAgICAgKi8NCiAgICBARGVwcmVjYXRlZA0KICAgIHB1YmxpYyBzdGF0aWMgY2xh c3MgTGluZVJlYWRlciBleHRlbmRzIG9yZy5hcGFjaGUuaGFkb29wLnV0aWwuTGluZVJlYWRlciB7 DQogICAgICAgIExpbmVSZWFkZXIoSW5wdXRTdHJlYW0gaW4pIHsNCiAgICAgICAgICAgIHN1cGVy KGluKTsNCiAgICAgICAgfQ0KDQogICAgICAgIExpbmVSZWFkZXIoSW5wdXRTdHJlYW0gaW4sIGlu dCBidWZmZXJTaXplKSB7DQogICAgICAgICAgICBzdXBlcihpbiwgYnVmZmVyU2l6ZSk7DQogICAg ICAgIH0NCg0KICAgICAgICBwdWJsaWMgTGluZVJlYWRlcihJbnB1dFN0cmVhbSBpbiwgQ29uZmln dXJhdGlvbiBjb25mKQ0KICAgICAgICAgICAgICAgIHRocm93cyBJT0V4Y2VwdGlvbiB7DQogICAg ICAgICAgICBzdXBlcihpbiwgY29uZik7DQogICAgICAgIH0NCiAgICB9DQoNCiAgICBwdWJsaWMg TXVsdGlwbGVMaW5lVGV4dFJlY29yZFJlYWRlcihDb25maWd1cmF0aW9uIGpvYiwgRmlsZVNwbGl0 IHNwbGl0KQ0KICAgICAgICAgICAgdGhyb3dzIElPRXhjZXB0aW9uIHsNCiAgICAgICAgdGhpcy5t YXhMaW5lTGVuZ3RoID0gam9iLmdldEludCgibWFwcmVkLmxpbmVyZWNvcmRyZWFkZXIubWF4bGVu Z3RoIiwNCiAgICAgICAgICAgICAgICBJbnRlZ2VyLk1BWF9WQUxVRSk7DQogICAgICAgIHRoaXMu bGluZXNOdW1iZXIgPSBqb2IuZ2V0SW50KCJtYXByZWQudGV4dHJlY29yZHJlYWRlci5saW5lY291 bnQiLA0KICAgICAgICAgICAgICAgIERFRkFVTFRfTElORV9OVU1CRVIpOw0KICAgICAgICBzdGFy dCA9IHNwbGl0LmdldFN0YXJ0KCk7DQogICAgICAgIGVuZCA9IHN0YXJ0ICsgc3BsaXQuZ2V0TGVu Z3RoKCk7DQogICAgICAgIGZpbmFsIFBhdGggZmlsZSA9IHNwbGl0LmdldFBhdGgoKTsNCiAgICAg ICAgY29tcHJlc3Npb25Db2RlY3MgPSBuZXcgQ29tcHJlc3Npb25Db2RlY0ZhY3Rvcnkoam9iKTsN CiAgICAgICAgZmluYWwgQ29tcHJlc3Npb25Db2RlYyBjb2RlYyA9IGNvbXByZXNzaW9uQ29kZWNz LmdldENvZGVjKGZpbGUpOw0KDQogICAgICAgIC8vIG9wZW4gdGhlIGZpbGUgYW5kIHNlZWsgdG8g dGhlIHN0YXJ0IG9mIHRoZSBzcGxpdA0KICAgICAgICBGaWxlU3lzdGVtIGZzID0gZmlsZS5nZXRG aWxlU3lzdGVtKGpvYik7DQogICAgICAgIEZTRGF0YUlucHV0U3RyZWFtIGZpbGVJbiA9IGZzLm9w ZW4oc3BsaXQuZ2V0UGF0aCgpKTsNCiAgICAgICAgYm9vbGVhbiBza2lwRmlyc3RMaW5lID0gZmFs c2U7DQogICAgICAgIGlmIChjb2RlYyAhPSBudWxsKSB7DQogICAgICAgICAgICBpbiA9IG5ldyBM aW5lUmVhZGVyKGNvZGVjLmNyZWF0ZUlucHV0U3RyZWFtKGZpbGVJbiksIGpvYik7DQogICAgICAg ICAgICBlbmQgPSBMb25nLk1BWF9WQUxVRTsNCiAgICAgICAgfSBlbHNlIHsNCiAgICAgICAgICAg IGlmIChzdGFydCAhPSAwKSB7DQogICAgICAgICAgICAgICAgc2tpcEZpcnN0TGluZSA9IHRydWU7 DQogICAgICAgICAgICAgICAgLS1zdGFydDsNCiAgICAgICAgICAgICAgICBmaWxlSW4uc2Vlayhz dGFydCk7DQogICAgICAgICAgICB9DQogICAgICAgICAgICBpbiA9IG5ldyBMaW5lUmVhZGVyKGZp bGVJbiwgam9iKTsNCiAgICAgICAgfQ0KICAgICAgICBpZiAoc2tpcEZpcnN0TGluZSkgeyAvLyBz a2lwIGZpcnN0IGxpbmUgYW5kIHJlLWVzdGFibGlzaCAic3RhcnQiLg0KICAgICAgICAgICAgc3Rh cnQgKz0gaW4ucmVhZExpbmUobmV3IFRleHQoKSwgMCwgKGludCkgTWF0aC5taW4oDQogICAgICAg ICAgICAgICAgICAgIChsb25nKSBJbnRlZ2VyLk1BWF9WQUxVRSwgZW5kIC0gc3RhcnQpKTsNCiAg ICAgICAgfQ0KICAgICAgICB0aGlzLnBvcyA9IHN0YXJ0Ow0KICAgIH0NCg0KICAgIHB1YmxpYyBN dWx0aXBsZUxpbmVUZXh0UmVjb3JkUmVhZGVyKElucHV0U3RyZWFtIGluLCBsb25nIG9mZnNldCwg bG9uZyBlbmRPZmZzZXQsDQogICAgICAgICAgICBpbnQgbWF4TGluZUxlbmd0aCkgew0KICAgICAg ICB0aGlzLm1heExpbmVMZW5ndGggPSBtYXhMaW5lTGVuZ3RoOw0KICAgICAgICB0aGlzLmluID0g bmV3IExpbmVSZWFkZXIoaW4pOw0KICAgICAgICB0aGlzLnN0YXJ0ID0gb2Zmc2V0Ow0KICAgICAg ICB0aGlzLnBvcyA9IG9mZnNldDsNCiAgICAgICAgdGhpcy5lbmQgPSBlbmRPZmZzZXQ7DQogICAg fQ0KDQogICAgcHVibGljIE11bHRpcGxlTGluZVRleHRSZWNvcmRSZWFkZXIoSW5wdXRTdHJlYW0g aW4sIGxvbmcgb2Zmc2V0LCBsb25nIGVuZE9mZnNldCwNCiAgICAgICAgICAgIENvbmZpZ3VyYXRp b24gam9iKSB0aHJvd3MgSU9FeGNlcHRpb24gew0KICAgICAgICB0aGlzLm1heExpbmVMZW5ndGgg PSBqb2IuZ2V0SW50KCJtYXByZWQubGluZXJlY29yZHJlYWRlci5tYXhsZW5ndGgiLA0KICAgICAg ICAgICAgICAgIEludGVnZXIuTUFYX1ZBTFVFKTsNCiAgICAgICAgdGhpcy5saW5lc051bWJlciA9 IGpvYi5nZXRJbnQoIm1hcHJlZC50ZXh0cmVjb3JkcmVhZGVyLmxpbmVjb3VudCIsDQogICAgICAg ICAgICAgICAgREVGQVVMVF9MSU5FX05VTUJFUik7DQogICAgICAgIHRoaXMuaW4gPSBuZXcgTGlu ZVJlYWRlcihpbiwgam9iKTsNCiAgICAgICAgdGhpcy5zdGFydCA9IG9mZnNldDsNCiAgICAgICAg dGhpcy5wb3MgPSBvZmZzZXQ7DQogICAgICAgIHRoaXMuZW5kID0gZW5kT2Zmc2V0Ow0KICAgIH0N Cg0KICAgIHB1YmxpYyBMb25nV3JpdGFibGUgY3JlYXRlS2V5KCkgew0KICAgICAgICByZXR1cm4g bmV3IExvbmdXcml0YWJsZSgpOw0KICAgIH0NCg0KICAgIHB1YmxpYyBUZXh0IGNyZWF0ZVZhbHVl KCkgew0KICAgICAgICByZXR1cm4gbmV3IFRleHQoKTsNCiAgICB9DQoNCiAgICAvKiogUmVhZCBh IGdpdmVuIG51bWJlciBvZiBsaW5lcy4NCiAgICAgKiAgTnVtYmVyIG9mIHJlYWQgbGluZXMgbXVz dCBiZSBzcGVjaWZpZWQgaW50byB0aGUgcHJvcGVydHkNCiAgICAgKiAgbWFwcmVkLnRleHRyZWNv cmRyZWFkZXIubGluZWNvdW50DQogICAgICovDQogICAgcHVibGljIHN5bmNocm9uaXplZCBib29s ZWFuIG5leHQoTG9uZ1dyaXRhYmxlIGtleSwgVGV4dCB2YWx1ZSkNCiAgICAgICAgICAgIHRocm93 cyBJT0V4Y2VwdGlvbiB7DQogICAgICAgIGJvb2xlYW4ga2V5U2V0ID0gZmFsc2U7DQogICAgICAg IGludCBsaW5lc1RvUmVhZCA9IGxpbmVzTnVtYmVyOw0KICAgICAgICANCiAgICAgICAgdmFsdWUu Y2xlYXIoKTsgLy8gcmUtaW5pdGlhbGl6ZSB2YWx1ZQ0KICAgICAgICB3aGlsZSAoKHBvcyA8IGVu ZCkgJiYgKGxpbmVzVG9SZWFkID4gMCkpIHsNCiAgICAgICAgICAgIGxpbmVzVG9SZWFkLS07DQog ICAgICAgICAgICBpZiAoIWtleVNldCkgew0KICAgICAgICAgICAgICAgIGtleS5zZXQocG9zKTsN CiAgICAgICAgICAgICAgICBrZXlTZXQgPSB0cnVlOw0KICAgICAgICAgICAgfQ0KDQogICAgICAg ICAgICBpbnQgbmV3U2l6ZSA9IGluLnJlYWRMaW5lKGxpbmVSZWFkLCBtYXhMaW5lTGVuZ3RoLCBN YXRoLm1heCgoaW50KSBNYXRoDQogICAgICAgICAgICAgICAgICAgIC5taW4oSW50ZWdlci5NQVhf VkFMVUUsIGVuZCAtIHBvcyksIG1heExpbmVMZW5ndGgpKTsNCiAgICAgICAgICAgIGlmIChuZXdT aXplID09IDApIHsNCiAgICAgICAgICAgICAgICB2YWx1ZSA9IGxpbmVSZWFkOyANCiAgICAgICAg ICAgICAgIHJldHVybiBmYWxzZTsNCiAgICAgICAgICAgIH0NCiAgICAgICAgICAgIHBvcyArPSBu ZXdTaXplOw0KICAgICAgICAgICAgdmFsdWUuYXBwZW5kKGxpbmVSZWFkLmdldEJ5dGVzKCksIDAs IGxpbmVSZWFkLmdldExlbmd0aCgpKTsNCiAgICAgICAgICAgIC8vIFRPRE8gdG9vIGxvbmcgbGlu ZXMgYXJlIG5vdCB3ZWxsIGhhbmRsZWQuLi4NCiAgICAgICAgICAgIC8vIHNlZSBwcm9jZXNzaW5n IGRvbmUgaW4gb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLkxpbmVSZWNvcmRSZWFkZXINCiAgICAg ICAgICAgIC8vIHVzaW5nIGNvbmZpZ3VyYXRpb24gdmFyaWFibGUgbWFwcmVkLmxpbmVyZWNvcmRy ZWFkZXIubWF4bGVuZ3RoDQogICAgICAgICAgICBpZiAobGluZXNUb1JlYWQgPT0gMCkgeyAvLyBs aW5lc1RvUmVhZCBsaW5lcyBhcmUgcmVhZA0KICAgICAgICAgICAgICAgcmV0dXJuIHRydWU7DQog ICAgICAgICAgICB9IA0KICAgICAgICB9DQogICAgICAgIHJldHVybiBmYWxzZTsNCiAgICB9DQoN CiAgICAvKioNCiAgICAgKiBHZXQgdGhlIHByb2dyZXNzIHdpdGhpbiB0aGUgc3BsaXQNCiAgICAg Ki8NCiAgICBwdWJsaWMgZmxvYXQgZ2V0UHJvZ3Jlc3MoKSB7DQogICAgICAgIGlmIChzdGFydCA9 PSBlbmQpIHsNCiAgICAgICAgICAgIHJldHVybiAwLjBmOw0KICAgICAgICB9IGVsc2Ugew0KICAg ICAgICAgICAgcmV0dXJuIE1hdGgubWluKDEuMGYsIChwb3MgLSBzdGFydCkgLyAoZmxvYXQpIChl bmQgLSBzdGFydCkpOw0KICAgICAgICB9DQogICAgfQ0KDQogICAgcHVibGljIHN5bmNocm9uaXpl ZCBsb25nIGdldFBvcygpIHRocm93cyBJT0V4Y2VwdGlvbiB7DQogICAgICAgIHJldHVybiBwb3M7 DQogICAgfQ0KDQogICAgcHVibGljIHN5bmNocm9uaXplZCB2b2lkIGNsb3NlKCkgdGhyb3dzIElP RXhjZXB0aW9uIHsNCiAgICAgICAgaWYgKGluICE9IG51bGwpIHsNCiAgICAgICAgICAgIGluLmNs b3NlKCk7DQogICAgICAgIH0NCiAgICB9DQp9DQoNCg== --_003_8DC8587EB3873A4C912B5A9519750B812080B08214PUEXCB1Fnante_--