Can't write binary file in python

Question

I need to write a binary file in python where first few lines are 8 byte ints and the rest a bunch of double precision numbers. How do I best do it? I tried this code, but I got stuck, becuase for some reason, this code generates only text files despite being open in wb.

if binary:
            outfile = open(filename,"wb")
            text = f"1\n8\n{nrcells}\n{nrspecs}\n"
            outfile.write(text.encode(encoding="UTF-8"))
            if self.rho.ndim == 3:
                for x3 in range(self.par["rho_dim"][2]):
                    for x2 in range(self.par["rho_dim"][1]):
                        text="\n".join(f"{d:.8e}" for d in self.rho[:, x2, x3]*10**(-10))+"\n"
                        outfile.write(text.encode(encoding="UTF-8"))

What exactly are you expecting as file contents? You are explicitly writing UTF-8 encoded text. Do you expect your 8 byte integers to occupy 8 bytes in the file? Adding newlines into the mix doesn't make sense in that case, because one of the bytes may contain the decimal value 10, which corresponds to a newline, effectively breaking your file format. — Michilus
– Michilus, Commented Aug 27, 2020 at 11:20

Michilus · Accepted Answer · 2020-08-27 11:56:09Z

If you want to write binary data, it's not enough to just open the file in binary mode.

When you are executing these lines

text = f"1\n8\n{nrcells}\n{nrspecs}\n"
outfile.write(text.encode(encoding="UTF-8"))

you are still writing textual data to it.

Instead, you may want to have a look at the struct and array standard library modules. They enable you to write your numerical data to bytes objects or files in binary modes.

To write your 1, 8, nrcells, nrspecs header, assuming they are 8 byte integers, you'd do something like this:

# q means "long long", or 8 bytes signed integer
header_bytes_content = struct.pack('qqqq', 1, 8, nrcells, nrspecs)  
outfile.write(header_bytes_content)

To write your float values, you could either write them one by one:

# d means 8 byte floating point; f means 4 byte floating point
float_bytes_content = struct.pack('d', float_value)
outfile.write(float_bytes_content)

or gather all the values into an array and write them to the file:

float_array = array.array('d', float_values)
float_array.tofile(outfile)

To read it back out of the file, you can use struct.unpack and array.fromfile

Note that, if you don't need the file to be binary, you probably shouldn't use a binary format. Textual formats are easier to read and write by code and humans, and usually don't take up much more space.

Collectives™ on Stack Overflow

Can't write binary file in python

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related