Return to Answer

With any Bourne-like shell:

{
  cat < bigfile | grep -v to-exclude
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> bigfile

For some reason, it seems people tend to forget about that 40 year old¹ and standard read+write redirection operator.

We open bigfile in read+write mode and (what matters most here) without truncation on stdout while bigfile is open (separately) on cat's stdin. After grep has terminated, and if it has removed some lines, stdout now points somewhere within bigfile, we need to get rid of what's beyond this point. Hence the perl command that truncates the file (truncate STDOUT) at the current position (as returned by tell STDOUT).

(the cat is for GNU grep that otherwise complains if stdin and stdout point to the same file).

^{¹ Well, while <> has been in the Bourne shell from the start in the late seventies, it was initially undocumented and not properly implemented. It was not in the original implementation of ash from 1989 and, while it is a POSIX sh redirection operator (since the early 90s as POSIX sh is based on ksh88 which always had it), it was not added to FreeBSD sh for instance until 2000, so portably 15 year old is probably more accurate. Also note that the default file descriptor when not specified is <> in all shells, except that in ksh93 it changed from 0 to 1 in ksh93t+ in 2010 (breaking backward compatibility and POSIX compliance)}

With any Bourne-like shell:

{
  cat < bigfile | grep -v to-exclude
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> bigfile

For some reason, it seems people tend to forget about that 40 year old¹ and standard read+write redirection operator.

We open bigfile in read+write mode and (what matters most here) without truncation on stdout while bigfile is open (separately) on cat's stdin. After grep has terminated, and if it has removed some lines, stdout now points somewhere within bigfile, we need to get rid of what's beyond this point. Hence the perl command that truncates the file (truncate STDOUT) at the current position (as returned by tell STDOUT).

(the cat is for GNU grep that otherwise complains if stdin and stdout point to the same file).

^{¹ Well, while <> has been in the Bourne shell from the start in the late seventies, it was initially undocumented and not properly implemented. It was not in the original implementation of ash from 1989 and, while it is a POSIX sh redirection operator (since the early 90s as POSIX sh is based on ksh88 which always had it), it was not added to FreeBSD sh for instance until 2000, so portably 15 year old is probably more accurate. Also note that the default file descriptor when not specified is 0 in all shells, except that in ksh93 it changed from 0 to 1 in ksh93t+ in 2010 (breaking backward compatibility and POSIX compliance)}

With any Bourne-like shell:

{
  cat < bigfile | grep -v to-exclude
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> bigfile

For some reason, it seems people tend to forget about that 40 year old¹ and standard read+write redirection operator.

We open bigfile in read+write mode and (what matters most here) without truncation on stdout while bigfile is open (separately) on cat's stdin. After grep has terminated, and if it has removed some lines, stdout now points somewhere within bigfile, we need to get rid of what's beyond this point. Hence the perl command that truncates the file (truncate STDOUT) at the current position (as returned by tell STDOUT).

(the cat is for GNU grep that otherwise complains if stdin and stdout point to the same file).

^{¹ Well, while <> has been in the Bourne shell from the start in the late seventies, it was initially undocumented and not properly implemented. It was not in the original implementation of ash from 1989 and, while it is a POSIX sh redirection operator (since the early 90s as POSIX sh is based on ksh88 which always had it), it was not added to FreeBSD sh for instance until 2000, so portably 15 year old is probably more accurate}

With any Bourne-like shell:

{
  cat < bigfile | grep -v to-exclude
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> bigfile

For some reason, it seems people tend to forget about that 40 year old¹ and standard read+write redirection operator.

We open bigfile in read+write mode and (what matters most here) without truncation on stdout while bigfile is open (separately) on cat's stdin. After grep has terminated, and if it has removed some lines, stdout now points somewhere within bigfile, we need to get rid of what's beyond this point. Hence the perl command that truncates the file (truncate STDOUT) at the current position (as returned by tell STDOUT).

(the cat is for GNU grep that otherwise complains if stdin and stdout point to the same file).

^{¹ Well, while <> has been in the Bourne shell from the start in the late seventies, it was initially undocumented and not properly implemented. It was not in the original implementation of ash from 1989 and, while it is a POSIX sh redirection operator (since the early 90s as POSIX sh is based on ksh88 which always had it), it was not added to FreeBSD sh for instance until 2000, so portably 15 year old is probably more accurate. Also note that the default file descriptor when not specified is <> in all shells, except that in ksh93 it changed from 0 to 1 in ksh93t+ in 2010 (breaking backward compatibility and POSIX compliance)}

With any Bourne-like shell:

{
  cat < bigfile | grep -v to-exclude
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> bigfile

For some reason, it seems people tend to forget about that 36 year old and standard read+write redirection operator.

We open bigfile in read+write mode and (what matters most here) without truncation on stdout while bigfile is open (separately) on cat's stdin. After grep has terminated, and if it has removed some lines, stdout now points somewhere within bigfile, we need to get rid of what's beyond this point. Hence the perl command that truncates the file (truncate STDOUT) at the current position (as returned by tell STDOUT).

(the cat is for GNU grep that otherwise complains if stdin and stdout point to the same file).

With any Bourne-like shell:

{
  cat < bigfile | grep -v to-exclude
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> bigfile

For some reason, it seems people tend to forget about that 40 year old¹ and standard read+write redirection operator.

We open bigfile in read+write mode and (what matters most here) without truncation on stdout while bigfile is open (separately) on cat's stdin. After grep has terminated, and if it has removed some lines, stdout now points somewhere within bigfile, we need to get rid of what's beyond this point. Hence the perl command that truncates the file (truncate STDOUT) at the current position (as returned by tell STDOUT).

(the cat is for GNU grep that otherwise complains if stdin and stdout point to the same file).

^{¹ Well, while <> has been in the Bourne shell from the start in the late seventies, it was initially undocumented and not properly implemented. It was not in the original implementation of ash from 1989 and, while it is a POSIX sh redirection operator (since the early 90s as POSIX sh is based on ksh88 which always had it), it was not added to FreeBSD sh for instance until 2000, so portably 15 year old is probably more accurate}