bash: Split binary file at predefined positions

Question

I have binary files which contain data structures of various length. I would like to save these blocks of data into separate files. The size of each block is known. The split command can, well, split a file but it does not stop after the first block of data. It slices the file into pieces of equal size.

Therefore, my current solution is to split and cat the remainder of the file back together, iterating my way through the data. This is very clumsy and may even fail in certain circumstances.

What is the best way to slice a binary file precisely at certain positions?

If I see it correctly, dd lets me copy the first block of arbitrary size, but then I can skip only at multiples within the input and output files, respectively. I would need offset and block size as independant parameters. — Metalbeard
– Metalbeard, Commented Mar 4, 2015 at 15:28

Mark Setchell · Accepted Answer · 2015-03-06 09:24:12Z

15

You can use two independent dd commands. One to seek arbitrarily, and another to copy arbitrary lengths.

SEEK=501
BYTES=387
dd if=yourfile bs=$SEEK skip=1 | dd bs=$BYTES count=1 > lump.bin

Note: Although counter-intuitive to what you are actually try to do, keep the blocksize high and the count low for best performance. What I mean is, if you want 8192 bytes, use bs=8192 count=1 rather than bs=1 count=8192.

edited Mar 6, 2015 at 9:24

answered Mar 4, 2015 at 16:29

Mark Setchell

211k32 gold badges310 silver badges507 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Metalbeard Over a year ago

Thank you. It solves my problem. And it is not counter-intuitive at all.

Janaka Bandara Over a year ago

Excellent! Saved a hell of a lot of time in my case

Pratyush Choudhary · Accepted Answer · 2023-01-09 19:39:25Z

I would need offset and block size as independant parameters.

Using the iflag operand may help you here. It is available in dd (coreutils) since 2012, and you can specify different units for bs, skip and count.

Note that skip and count must either be in bytes or match the bs value.

An example usage will be:

dd if=infile.bin of=outfile.bin bs=4K skip=1234 count=20 iflag=skip_bytes,count_bytes

which would copy 20 bytes from input file, starting from byte 1235.

You can refer here for more detailed usage: https://askubuntu.com/a/1178771

Collectives™ on Stack Overflow

bash: Split binary file at predefined positions

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related