commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Albretch Mueller <lbrt...@gmail.com>
Subject Re: [compress] BZip2CompressorInputStream stops working without rhyme or reason ...
Date Wed, 14 Oct 2020 10:21:49 GMT
 I don't know what could there apaprently be exactly at byte offset
2848 in some buffer but files reporing to be fine by bzip2 --test
can't be processed by BZip2CompressorInputStream:
~
$ _IFL="/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream1.xml-p1p41242.bz2"
$ ls -l "${_IFL}"
-r--r--r-- 1 lbrtchx lbrtchx 242624781 Sep 22 05:40
/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream1.xml-p1p41242.bz2
$ file --brief "${_IFL}"
bzip2 compressed data, block size = 900k
$ time bzip2 --test --verbose "${_IFL}"
  /home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream1.xml-p1p41242.bz2:
ok

real    2m0.650s
user    2m0.076s
sys     0m0.256s

$ _IFL="/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream4.xml-p311330p558391.bz2"
$ ls -l "${_IFL}"
-r--r--r-- 1 lbrtchx lbrtchx 394001572 Sep 22 05:49
/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream4.xml-p311330p558391.bz2
$ file --brief "${_IFL}"
bzip2 compressed data, block size = 900k
$ time bzip2 --test --verbose  "${_IFL}"
  /home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream4.xml-p311330p558391.bz2:
ok

real    3m6.249s
user    3m5.192s
sys     0m0.628s

$ _IFL="/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream5.xml-p558392p958045.bz2"
$ ls -l "${_IFL}"
-r--r--r-- 1 lbrtchx lbrtchx 427323881 Sep 22 05:51
/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream5.xml-p558392p958045.bz2
$ file --brief "${_IFL}"
bzip2 compressed data, block size = 900k
$ time bzip2 --test --verbose  "${_IFL}"
  /home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream5.xml-p558392p958045.bz2:
ok

real    3m20.861s
user    3m19.296s
sys     0m0.988s

$ _IFL="/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream6.xml-p958046p1483661.bz2"
$ ls -l "${_IFL}"
-r--r--r-- 1 lbrtchx lbrtchx 458830618 Sep 22 05:52
/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream6.xml-p958046p1483661.bz2
$ file --brief "${_IFL}"
bzip2 compressed data, block size = 900k
$ time bzip2 --test --verbose  "${_IFL}"
  /home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream6.xml-p958046p1483661.bz2:
ok

real    3m34.213s
user    3m32.636s
sys     0m1.056s
$


$ _IFL="/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/logs/UnKmprssBZ2_02Test_20201013234903.log"
$ tail -n 10 "${_IFL}"
// __ Files Context of |4| files containing a total of |1522780852| bytes!
// __ [0/4): ...(30.131%)
|/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream6.xml-p958046p1483661.bz2|
// __ aOFlNm: |/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/REF/enwiki-20200920-pages-articles-multistream6-p958046p1483661.xml|
// __ |2848|2848|java.io.IOException:
// __ Read bytes and file lenght not the same! lTtlRdByts: |2848|
(lTtlRdByts != lFlL),  lFlL: |458830618|
        at UnKmprssBZ2_02Test.main(UnKmprssBZ2_02Test.java:254)

real    0m1.759s
user    0m2.920s
sys     0m0.196s

$ _IFL="/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/logs/UnKmprssBZ2_02Test_20201013234826.log"
$ tail -n 10 "${_IFL}"
// __ Files Context of |4| files containing a total of |1522780852| bytes!
// __ [0/4): ...(28.062%)
|/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream5.xml-p558392p958045.bz2|
// __ aOFlNm: |/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/REF/enwiki-20200920-pages-articles-multistream5-p558392p958045.xml|
// __ |2848|2848|java.io.IOException:
// __ Read bytes and file lenght not the same! lTtlRdByts: |2848|
(lTtlRdByts != lFlL),  lFlL: |427323881|
        at UnKmprssBZ2_02Test.main(UnKmprssBZ2_02Test.java:254)

real    0m1.669s
user    0m2.720s
sys     0m0.220s

$ _IFL="/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/logs/UnKmprssBZ2_02Test_20201013234708.log"
$ tail -n 10 "${_IFL}"
// __ Files Context of |4| files containing a total of |1522780852| bytes!
// __ [0/4): ...(25.874%)
|/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream4.xml-p311330p558391.bz2|
// __ aOFlNm: |/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/REF/enwiki-20200920-pages-articles-multistream4-p311330p558391.xml|
// __ |2848|2848|java.io.IOException:
// __ Read bytes and file lenght not the same! lTtlRdByts: |2848|
(lTtlRdByts != lFlL),  lFlL: |394001572|
        at UnKmprssBZ2_02Test.main(UnKmprssBZ2_02Test.java:254)

real    0m1.665s
user    0m2.752s
sys     0m0.172s

$ _IFL="/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/logs/UnKmprssBZ2_02Test_20201013234602.log"
$ tail -n 10 "${_IFL}"
// __ Files Context of |4| files containing a total of |1522780852| bytes!
// __ [0/4): ...(15.933%)
|/home/lbrtchx/cmllpz/LklWb/org/wikimedia/dumps/enwiki/20200920/enwiki-20200920-pages-articles-multistream1.xml-p1p41242.bz2|
// __ aOFlNm: |/home/lbrtchx/cmllpz/prjx/kd/java/IO/compress/REF/enwiki-20200920-pages-articles-multistream1-p1p41242.xml|
// __ |2848|2848|java.io.IOException:
// __ Read bytes and file lenght not the same! lTtlRdByts: |2848|
(lTtlRdByts != lFlL),  lFlL: |242624781|
        at UnKmprssBZ2_02Test.main(UnKmprssBZ2_02Test.java:254)

real    0m1.691s
user    0m2.756s
sys     0m0.216s
$

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message