avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris D'Urso (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AVRO-1486) avro-tools-1.7.6.jar fromtext to stdout fails because status also printed to stdout
Date Mon, 24 Mar 2014 22:33:45 GMT
Chris D'Urso created AVRO-1486:
----------------------------------

             Summary: avro-tools-1.7.6.jar fromtext  to stdout fails because status also printed
to stdout
                 Key: AVRO-1486
                 URL: https://issues.apache.org/jira/browse/AVRO-1486
             Project: Avro
          Issue Type: Bug
          Components: java
    Affects Versions: 1.7.6
            Reporter: Chris D'Urso


[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case, the creation
of test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  test.txt.avro
Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes
where applicable
Writing line = 5
Writing line = 5
Writing line = 7
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
-rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
avro.codec
snappy
avro.schema
"bytes"
f@5:B
one	1
two	2
three	3
f@5:B
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
104 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$  # now what should be equivalent is not
so
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  - > test2.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line" is absent, and
this is because its been written to stdout as well as the true payload
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
avro.codec
snappy
avro.schema
"bytes"
gD?Writing line = 5
Writing line = 5
Writing line = 7
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro can be restored
but not test2.txt.avro, plus test2 is bigger due to having mixed stream data
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid
sync!
	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
	at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
	at org.apache.avro.tool.Main.run(Main.java:84)
	at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Invalid sync!
	at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
	... 3 more
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
155



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message