3

I have been using python to do ascii-to-binary translations and kept running into issues with parsing the result. Eventually I thought to look at what the Python commands were generating.

There seems to be a rouge 0xc2 inserted in the output (for example):

$ python -c 'print("\x80")' | xxd
00000000: c280 0a                                  ...

Indeed this happens regardless of where such bytes are used:

$ python -c 'print("Test\x80Test2\x81")' | xxd
00000000: 5465 7374 c280 5465 7374 32c2 810a       Test..Test2...

On a hunch, I poked around at UTF-8 and sure enough, U+0080 is encoded as 0xc2 0x80. Apparently, Python takes the liberty of assuming by \x80 I actually meant the encoding for U+0080. Is there a way to change this default behavior or otherwise explicitly dictate my intention of including the singlar byte 0x80 and not the UTF encoding?

Python 3.6.2

Improve this question
1

2 Answers 2

Reset to default
5

Python 3 does the right thing of inserting a character into a str which is string of characters, not a byte sequence.

UTF8 is the default encoding. If you need to insert a byte, a different encoding where that character is represented as a byte is needed.

$ PYTHONIOENCODING=iso-8859-1 python3 -c 'print("\x80")' | xxd
00000000: 800a

PYTHONIOENCODING

If this is set before running the interpreter, it overrides the encoding used for stdin/stdout/stderr, in the syntax encodingname:errorhandler. Both the encodingname and the :errorhandler parts are optional and have the same meaning as in str.encode().

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Changing the global encoding for strings is a (quite inelegant) global solution to a local problem. If OP is building a bytes sequence and not a character sequence he should use byte objects and not str objects (or, of anything, encode just the relevant str objects to Latin1 using the encode method).
@MatteoItalia Yes, but I do not understand this case like a "local problem". The OP asked for changing the default and mentioned that ascii-to-binary translations are what he is using the program for.
Fair enough, although the OP isn't really doing ascii-to-binary translation, since b'\x80' isn't legal ASCII.
@PM2Ring, Such pendantism. Sure, \x80 is not legal printable ASCII, but it was clear what my intentions were. Perhaps hex to binary is a better description.
@sherrellbc Perhaps it is a bit pedantic, but when you're doing stuff with character encodings it pays to be pedantic. ;)
4

If you want to output raw bytes in Python 3 you shouldn't be using the print function, since it's for outputting text in your default encoding. Instead, you can use sys.stdout.buffer.write.

ASCII is a 7 bit encoding, so if your so-called ASCII contains characters like b'\x80' it's not legal ASCII. Perhaps your data is actually encoded with iso-8859-1, aka latin-1, or it could be the closely-related Windows variant cp1252. To do this kind of thing correctly you need to determine the actual encoding that was used to create the data.

If you want to output "Test\x80Test2\x81" and have the hex dump look like this:

00000000  54 65 73 74 80 54 65 73  74 32 81                 |Test.Test2.|

You can do

import sys
s = "Test\x80Test2\x81"
sys.stdout.buffer.write(s.encode('latin1'))

This works because Latin-1 is a subset of Unicode. Here's a quick demo:

import binascii

a = ''.join([chr(i) for i in range(256)])
b = a.encode('latin1')
print(binascii.hexlify(b))

output

b'000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff'

However, if you're actually working with binary data then you shouldn't be storing it in text strings in the first place, you should be using bytes, or possibly bytearray. The sane way to produce the b bytes string from my previous example is to do 

b = bytes(range(256))

And if you have a bytes object like b"Test\x80Test2\x81" you can dump those bytes to stdout with 

sys.stdout.buffer.write(b"Test\x80Test2\x81")
Improve this answer

3 Comments

Thanks for the example. There is no data storage. My use-case was to generate a string of ASCII and non-printables and then pipe it directly to stdin of a program under test.
@sherrellbc No worries. In that case you definitely don't want to use text strings for this, do everything with bytes and bytearrays.
@sherrellbc And if the program that you're piping this binary data to is a Python 3 program then you should get it to read the data from sys.stdin.buffer, don't read it by calling input, because that expects text strings.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.