Return-Path: X-Original-To: apmail-commons-dev-archive@www.apache.org Delivered-To: apmail-commons-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3C7AF9A46 for ; Sun, 11 Mar 2012 14:05:36 +0000 (UTC) Received: (qmail 87616 invoked by uid 500); 11 Mar 2012 14:05:35 -0000 Delivered-To: apmail-commons-dev-archive@commons.apache.org Received: (qmail 87517 invoked by uid 500); 11 Mar 2012 14:05:35 -0000 Mailing-List: contact dev-help@commons.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Commons Developers List" Delivered-To: mailing list dev@commons.apache.org Received: (qmail 87507 invoked by uid 99); 11 Mar 2012 14:05:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Mar 2012 14:05:35 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [212.27.42.6] (HELO smtp6-g21.free.fr) (212.27.42.6) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Mar 2012 14:05:29 +0000 Received: from [192.168.0.10] (unknown [82.247.19.29]) by smtp6-g21.free.fr (Postfix) with ESMTP id 5F8AF822A6 for ; Sun, 11 Mar 2012 15:05:01 +0100 (CET) Message-ID: <4F5CB10C.507@apache.org> Date: Sun, 11 Mar 2012 15:05:00 +0100 From: Emmanuel Bourg User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.27) Gecko/20120216 Thunderbird/3.1.19 MIME-Version: 1.0 To: Commons Developers List Subject: [csv] Performance comparison Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms060709040203080003010406" X-Virus-Checked: Checked by ClamAV on apache.org --------------ms060709040203080003010406 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Hi, I compared the performance of Commons CSV with the other CSV parsers=20 available. I took the world cities file from Maxmind as a test file [1], = it's a big file of 130M with 2.8 million records. Here are the results obtained on a Core 2 Duo E8400 after several=20 iterations to let the JIT compiler kick in: Direct read 750 ms Java CSV 3328 ms Super CSV 3562 ms (+7%) OpenCSV 3609 ms (+8.4%) GenJava CSV 3844 ms (+15.5%) Commons CSV 4656 ms (+39.9%) Skife CSV 4813 ms (+44.6%) I also tried Nuiton CSV and Esperio CSV but I couldn't figure how to use = them. I haven't analyzed why Commons CSV is slower yet, but it seems there is=20 room for improvements. The memory usage will have to be compared too,=20 I'm looking for a way to measure it. Emmanuel Bourg [1] http://www.maxmind.com/download/worldcities/worldcitiespop.txt.gz --------------ms060709040203080003010406 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIUDTCC BJ0wggOFoAMCAQICEDQ96SusJzT/j8s0lPvMcFQwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UE BhMCU0UxFDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0w NTA2MDcwODA5MTBaFw0yMDA1MzAxMDQ4MzhaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMC VVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5l dHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVRO LVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVN NRm5pELlzkniii8efNIxB8dOtINknS4p1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQy lbsMTzC9mKALi+VuG6JG+ni8om+rWV6lL8/K2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXq vgvOdjp6Dpvq/NonWz1zHyLmSGHGTPNpsaguG7bUMSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6 hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7NlyP0e03RiqhjKaJMeoYV+9Udly/hNVyh00jT/MLbu 9mIwFIws6wIDAQABo4H0MIHxMB8GA1UdIwQYMBaAFK29mHo0tCb3+sQmVO8DveAky1QaMB0G A1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/ BAUwAwEB/zARBgNVHSAECjAIMAYGBFUdIAAwRAYDVR0fBD0wOzA5oDegNYYzaHR0cDovL2Ny bC51c2VydHJ1c3QuY29tL0FkZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMDUGCCsGAQUFBwEB BCkwJzAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0B AQUFAAOCAQEAAbyc42MosPMxAcLfe91ioAGdIzEPnJJzU1HqH0z61p/Eyi9nfngzD3QWuZGH kfWKJvpkcADYHvkLBGJQh5OB1Nr1I9s0u4VWtHA0bniDNx6FHMURFZJfhxe9rGr98cLRzIlf sXzwPlHyNfN87GCYazor4O/fs32G67Ub9VvsonyYE9cAULnRLXPeA3h04QWFMV7LmrmdlMa5 lDd1ctxE+2fo8PolHlKn2iXpR+CgxzygTrEKNvt3SJ/vl4r7tP7jlBSog7xcLT/SYHFg7sJx ggzpiDbj2iC0o6BsqpZLuICOdcpJB/Y7FLrf3AXZn9vgsuZNoHgm5+ctbn9fxh6IFTCCBRow ggQCoAMCAQICEG0Z6qcZT2ozIuYiMnqqcd4wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYT AlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRo ZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29t MTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h aWwwHhcNMTEwNDI4MDAwMDAwWhcNMjAwNTMwMTA0ODM4WjCBkzELMAkGA1UEBhMCR0IxGzAZ BgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMR Q09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0aGVudGljYXRp b24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AJKEhFtLV5jUXi+LpOFAyKNTWF9mZfEyTvefMn1V0HhMVbdClOD5J3EHxcZppLkyxPFAGpDM J1Zifxe1cWmu5SAb5MtjXmDKokH2auGj/7jfH0htZUOMKi4rYzh337EXrMLaggLW1DJq1Gdv IBOPXDX65VSAr9hxCh03CgJQU2yVHakQFLSZlVkSMf8JotJM3FLb3uJAAVtIaN3FSrTg7SQf Oq9xXwfjrL8UO7AlcWg99A/WF1hGFYE8aIuLgw9teiFX5jSw2zJ+40rhpVJyZCaRTqWSD//g sWD9Gm9oUZljjRqLpcxCm5t9ImPTqaD8zp6Q30QZ9FxbNboW86eb/8ECAwEAAaOCAUswggFH MB8GA1UdIwQYMBaAFImCZ33EnSZwAEu0UEh83j2uBG59MB0GA1UdDgQWBBR6E04AdFvGeGNk J8Ev4qBbvHnFezAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAE CjAIMAYGBFUdIAAwWAYDVR0fBFEwTzBNoEugSYZHaHR0cDovL2NybC51c2VydHJ1c3QuY29t L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwdAYIKwYB BQUHAQEEaDBmMD0GCCsGAQUFBzAChjFodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVROQWRk VHJ1c3RDbGllbnRfQ0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1c3Qu Y29tMA0GCSqGSIb3DQEBBQUAA4IBAQCF1r54V1VtM39EUv5C1QaoAQOAivsNsv1Kv/avQUn1 G1rF0q0bc24+6SZ85kyYwTAo38v7QjyhJT4KddbQPTmGZtGhm7VNm2+vKGwdr+XqdFqo2rHA 8XV6L566k3nK/uKRHlZ0sviN0+BDchvtj/1gOSBH+4uvOmVIPJg9pSW/ve9g4EnlFsjrP0OD 8ODuDcHTzTNfm9C9YGqzO/761Mk6PB/tm/+bSTO+Qik5g+4zaS6CnUVNqGnagBsePdIaXXxH maWbCG0SmYbWXVcHG6cwvktJRLiQfsrReTjrtDP6oDpdJlieYVUYtCHVmdXgQ0BCML7qpeeU 0rD+83X5f27nMIIFIzCCBAugAwIBAgIQOtEfOp+FCOCSgeJgvKa5DDANBgkqhkiG9w0BAQUF ADCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE BxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9E TyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTAeFw0xMTEwMjYw MDAwMDBaFw0xMjEwMjUyMzU5NTlaMCIxIDAeBgkqhkiG9w0BCQEWEWVib3VyZ0BhcGFjaGUu b3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0fz4jOn5sMktk6eRpRSXe/uM 6NmsQ+3vo9jQewKuKH8OEr77tHv6bPZljukFyIjTa1hQaptRTOFVHlh24gzUDSsLIMBASZcH KkLy0FqkGCEqoE1jtVGUFT+pYwq5l+seXYOyA8HnaTpeNgnP7qt7V/4q0EUJS5072qZdA9cF yLKpIUTuEyfzNHlJSVZT5PbBI8SXm5ROT9735mXyjHNs80zPSxbgCgySdnVk7JCWakF5s2eC sozCMDnFhza8sP2zfUfWvOrMeon3tqHwOQIasUtTN/x3k5KatxtwFJN1iFZItFD1ZDyUstql BEON9MoKqqAxMPU+nEnvJfwwBnS5AwIDAQABo4IB4TCCAd0wHwYDVR0jBBgwFoAUehNOAHRb xnhjZCfBL+KgW7x5xXswHQYDVR0OBBYEFPojKbFykDWFQo+0JisqdbMHdcLbMA4GA1UdDwEB /wQEAwIFoDAMBgNVHRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMF AjARBglghkgBhvhCAQEEBAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggr BgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwVwYDVR0fBFAwTjBMoEqg SIZGaHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09NT0RPQ2xpZW50QXV0aGVudGljYXRpb25h bmRTZWN1cmVFbWFpbENBLmNybDCBiAYIKwYBBQUHAQEEfDB6MFIGCCsGAQUFBzAChkZodHRw Oi8vY3J0LmNvbW9kb2NhLmNvbS9DT01PRE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3Vy ZUVtYWlsQ0EuY3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wHAYD VR0RBBUwE4ERZWJvdXJnQGFwYWNoZS5vcmcwDQYJKoZIhvcNAQEFBQADggEBAC7WwbS8wdOt SLsZ0BpXd/e+lXdHNJG9lD2FUvOEgLzsG5ZQC2lkVcUBStDx/nIq2URScmUI22hFGEEQpXh7 ZYes+iAoYctm9bRI21nOg/QXAG8rPL3wMlxN6TlQZpvh+qn5ke+fReEozrsd40vV+Earz8Sy l+dC8PIRYQE9ELRUoacZ8930chr4b+VH12/ScmCGbCHjBZ+/cv5egx2DlmllwA0MKRsuHZda vfwUbu7OJpVz/Pm0tPrR6gccPAwZRRcbQcO+zdIU+agLrJEP30BqYCkyDP9FZeQfFgHZPBYU xAVYBvCnZxq+8fzYYwvlCf6fmSfqnFo4kuh2V0Fq4GcwggUjMIIEC6ADAgECAhA60R86n4UI 4JKB4mC8prkMMA0GCSqGSIb3DQEBBQUAMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3Jl YXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0Eg TGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2Vj dXJlIEVtYWlsIENBMB4XDTExMTAyNjAwMDAwMFoXDTEyMTAyNTIzNTk1OVowIjEgMB4GCSqG SIb3DQEJARYRZWJvdXJnQGFwYWNoZS5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK AoIBAQDR/PiM6fmwyS2Tp5GlFJd7+4zo2axD7e+j2NB7Aq4ofw4Svvu0e/ps9mWO6QXIiNNr WFBqm1FM4VUeWHbiDNQNKwsgwEBJlwcqQvLQWqQYISqgTWO1UZQVP6ljCrmX6x5dg7IDwedp Ol42Cc/uq3tX/irQRQlLnTvapl0D1wXIsqkhRO4TJ/M0eUlJVlPk9sEjxJeblE5P3vfmZfKM c2zzTM9LFuAKDJJ2dWTskJZqQXmzZ4KyjMIwOcWHNryw/bN9R9a86sx6ife2ofA5AhqxS1M3 /HeTkpq3G3AUk3WIVki0UPVkPJSy2qUEQ430ygqqoDEw9T6cSe8l/DAGdLkDAgMBAAGjggHh MIIB3TAfBgNVHSMEGDAWgBR6E04AdFvGeGNkJ8Ev4qBbvHnFezAdBgNVHQ4EFgQU+iMpsXKQ NYVCj7QmKyp1swd1wtswDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwIAYDVR0lBBkw FwYIKwYBBQUHAwQGCysGAQQBsjEBAwUCMBEGCWCGSAGG+EIBAQQEAwIFIDBGBgNVHSAEPzA9 MDsGDCsGAQQBsjEBAgEBATArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8u bmV0L0NQUzBXBgNVHR8EUDBOMEygSqBIhkZodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01P RE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3JsMIGIBggrBgEFBQcB AQR8MHowUgYIKwYBBQUHMAKGRmh0dHA6Ly9jcnQuY29tb2RvY2EuY29tL0NPTU9ET0NsaWVu dEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6 Ly9vY3NwLmNvbW9kb2NhLmNvbTAcBgNVHREEFTATgRFlYm91cmdAYXBhY2hlLm9yZzANBgkq hkiG9w0BAQUFAAOCAQEALtbBtLzB061IuxnQGld3976Vd0c0kb2UPYVS84SAvOwbllALaWRV xQFK0PH+cirZRFJyZQjbaEUYQRCleHtlh6z6IChhy2b1tEjbWc6D9BcAbys8vfAyXE3pOVBm m+H6qfmR759F4SjOux3jS9X4RqvPxLKX50Lw8hFhAT0QtFShpxnz3fRyGvhv5UfXb9JyYIZs IeMFn79y/l6DHYOWaWXADQwpGy4dl1q9/BRu7s4mlXP8+bS0+tHqBxw8DBlFFxtBw77N0hT5 qAuskQ/fQGpgKTIM/0Vl5B8WAdk8FhTEBVgG8KdnGr7x/NhjC+UJ/p+ZJ+qcWjiS6HZXQWrg ZzGCBAwwggQIAgEBMIGoMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5j aGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE5 MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWls IENBAhA60R86n4UI4JKB4mC8prkMMAkGBSsOAwIaBQCgggI4MBgGCSqGSIb3DQEJAzELBgkq hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEyMDMxMTE0MDUwMFowIwYJKoZIhvcNAQkEMRYE FItudQlyu0yW5d62DXZIy6lXFpb6MF8GCSqGSIb3DQEJDzFSMFAwCwYJYIZIAWUDBAECMAoG CCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggq hkiG9w0DAgIBKDCBuQYJKwYBBAGCNxAEMYGrMIGoMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UE CBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01P RE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBh bmQgU2VjdXJlIEVtYWlsIENBAhA60R86n4UI4JKB4mC8prkMMIG7BgsqhkiG9w0BCRACCzGB q6CBqDCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G A1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENP TU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIQOtEfOp+F COCSgeJgvKa5DDANBgkqhkiG9w0BAQEFAASCAQC3baFOjiM8r53xbycXKXMj3QaQYeISsj3v HFAO8YhHaboXXZpNdZum72Y9mY2DCuyRCE6RaXTav2/ZM91BZvxMLABEqHpoEgM3hW2m/fSu LOb8rgXn8W+3QVYskWVjLFhA2BBNO0OM6ojmHJlRnI0I2zySaYSppXCU6NrGf9iojnL62CDJ k6OzW1uTrZMtbtwVhH4x0OwwelbeDk5oTXFydFcFuPZhLbtzLPcx1QRhxzFU6eRKtWXZMOyy KXc18EtBmxp9Y7o4xgvxrP/X5CD3BRcBvAlANwMfa17IkRu+q1IshmrC+e7GgJnE0x+5yKyu MB3fubvVQCMW9oFkHPJAAAAAAAAA --------------ms060709040203080003010406--