avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris D'Urso (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AVRO-1486) avro-tools-1.7.6.jar fromtext to stdout fails because status also printed to stdout
Date Tue, 25 Mar 2014 17:14:14 GMT

     [ https://issues.apache.org/jira/browse/AVRO-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris D'Urso updated AVRO-1486:
-------------------------------

    Description: 
Apparently unwanted status information is being printed out to standard output.  This interferes
with the '\-' option when attempting to direct aver encoded binary data to a pipe.  A possible
workaround is to use a named pipe instead, but really the program should either be quiet with
its status output at least in the case where stdout is the file handle directed by the '\-'
option.

{quote}
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case, the creation
of test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  test.txt.avro
Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes
where applicable
Writing line = 5
Writing line = 5
Writing line = 7
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
-rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
avro.codec
snappy
avro.schema
"bytes"
f@5:B
one	1
two	2
three	3
f@5:B
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
104 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$  # now what should be equivalent is not
so
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  - > test2.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line" is absent, and
this is because its been written to stdout as well as the true payload
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
avro.codec
snappy
avro.schema
"bytes"
gD?Writing line = 5
Writing line = 5
Writing line = 7
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro can be restored
but not test2.txt.avro, plus test2 is bigger due to having mixed stream data
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid
sync!
	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
	at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
	at org.apache.avro.tool.Main.run(Main.java:84)
	at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Invalid sync!
	at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
	... 3 more
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
155
{quote}

  was:
Apparently unwanted status information is being printed out to standard output.  This interferes
with the '\-' option when attempting to direct aver encoded binary data to a pipe.  A possible
workaround is to use a named pipe instead, but really the program should either be quiet with
its status output at least in the case where stdout is the file handle directed by the '\-'
option.


[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case, the creation
of test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  test.txt.avro
Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes
where applicable
Writing line = 5
Writing line = 5
Writing line = 7
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
-rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
avro.codec
snappy
avro.schema
"bytes"
f@5:B
one	1
two	2
three	3
f@5:B
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
104 test.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$  # now what should be equivalent is not
so
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  - > test2.txt.avro
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line" is absent, and
this is because its been written to stdout as well as the true payload
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
avro.codec
snappy
avro.schema
"bytes"
gD?Writing line = 5
Writing line = 5
Writing line = 7
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro can be restored
but not test2.txt.avro, plus test2 is bigger due to having mixed stream data
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
one	1
two	2
three	3
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid
sync!
	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
	at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
	at org.apache.avro.tool.Main.run(Main.java:84)
	at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Invalid sync!
	at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
	... 3 more
[b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
155


> avro-tools-1.7.6.jar fromtext  to stdout fails because status also printed to stdout
> ------------------------------------------------------------------------------------
>
>                 Key: AVRO-1486
>                 URL: https://issues.apache.org/jira/browse/AVRO-1486
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.7.6
>            Reporter: Chris D'Urso
>
> Apparently unwanted status information is being printed out to standard output.  This
interferes with the '\-' option when attempting to direct aver encoded binary data to a pipe.
 A possible workaround is to use a named pipe instead, but really the program should either
be quiet with its status output at least in the case where stdout is the file handle directed
by the '\-' option.
> {quote}
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge] # First the functioning case, the creation
of test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt
> one	1
> two	2
> three	3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  test.txt.avro
> Mar 24, 2014 6:13:50 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
> WARNING: Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
> Writing line = 5
> Writing line = 5
> Writing line = 7
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ ls -l test.txt.avro
> -rwxrwxrwx 1 b_ebayadvertising hdmi-ebayadvertising 104 Mar 24 18:13 test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test.txt.avro
> avro.codec
> snappy
> avro.schema
> "bytes"
> f@5:B
> one	1
> two	2
> three	3
> f@5:B
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ wc -c test.txt.avro
> 104 test.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$  # now what should be equivalent is
not so
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt | java -jar ~/lib/avro-tools-1.7.6.jar
fromtext --codec snappy -  - > test2.txt.avro
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that "Writing line" is absent,
and this is because its been written to stdout as well as the true payload
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ strings test2.txt.avro
> avro.codec
> snappy
> avro.schema
> "bytes"
> gD?Writing line = 5
> Writing line = 5
> Writing line = 7
> one	1
> two	2
> three	3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ # note that test.txt.avro can be restored
but not test2.txt.avro, plus test2 is bigger due to having mixed stream data
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
> one	1
> two	2
> three	3
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | java -jar ~/lib/avro-tools-1.7.6.jar
totext -  -
> Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException:
Invalid sync!
> 	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
> 	at org.apache.avro.tool.ToTextTool.run(ToTextTool.java:77)
> 	at org.apache.avro.tool.Main.run(Main.java:84)
> 	at org.apache.avro.tool.Main.main(Main.java:73)
> Caused by: java.io.IOException: Invalid sync!
> 	at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293)
> 	at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198)
> 	... 3 more
> [b_ebayadvertising@phx-ctd-hadoopas1 id_to_purge]$ cat test2.txt.avro | wc -c
> 155
> {quote}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message