3

I had a problem where I wanted to find the length of each path from a find command. My first attempt was to run something like this:

find . -exec sh -c "echo {} | wc -c" \;

I got this idea from this answer. (The command above is not my question, I'm just using it as an example and it's completely contrived. Also, I may need more than one pipe sometimes.)

But when I ran it, there were errors in the output, likely due to special characters in the output paths. Unfortunately, I don't have the knowledge to troubleshoot which paths caused the issue and the error messages weren't informative. Regardless...

I later stumbled upon this answer:

The find command executes the command directly. The command, including the filename argument, will not be processed by the shell or anything else that might modify the filename. It's very safe.

That seems very convenient. So convenient, in fact, that the -exec sh -c ... "cure" seems worse than the disease.

So my question is, what should I do when I need to pipe commands with find and my paths may have special characters? Is there a generic solution to this problem where I don't have to consider a bunch of caveats? I am using bash.

Note: This is a similar question: how best to send the output of a find + exec command to a pipeline? The difference is, I'm not necessarily trying to pipe the output outside of -exec. i.e., if find ... -exec ... foo {} | bar \; is the way to go, that's perfectly fine by me. I'm just looking for a generic path of least resistance, the structure of the command isn't important to me.

Improve this question

2 Answers 2

Reset to default
3

Pass the filename as an argument to the shell script:

find . -exec sh -c 'printf "%s\n" "$1" | wc -c' sh {} \;

or for multiple files per shell invocation:

find . -exec sh -c 'for x in "$@"; do printf "%s\n" "$x" | wc -c; done' sh {} +

Your command

find . -exec sh -c "echo {} | wc -c" \;

would insert the filename as-is on the shell command line. It'd only work for filenames that don't contain whitespace or characters that are special to the shell. E.g. something like a Don't stop me now.mp3, this&that.txt would cause problems. (The first would produce an unterminated quoted string, and the second would start echo in the background, and then try to run a command called that.txt.)

On the other hand, sh -c ... sh {} \; (or ... {} + has find pass the filename(s) as a distinct argument(s) to the shell, where they'll then be available in the positional parameters and can be used without them getting mixed with the shell syntax. ("$1" for the first, "$@" for the whole list. Remember the quotes.)

For the case of checking the filename length, you could also get it with "${#var}" in the shell, except that it gives the length in characters as per the current locale, while wc -c counts bytes.

Improve this answer
8
  • Maybe printf rather than echo, so you can handle paths starting with a dash, etc. (I know it's the contrived example from the question, but why not fix it too) Commented Apr 27, 2022 at 20:26
  • Thanks for answering. Can you explain why the first one works when paths have strange characters? Also, what do you mean by "multiple files per shell invocation"? Commented Apr 27, 2022 at 20:29
  • @DanielKaplan, see edit (I was just typing that). With echo {} | wc, the filename gets embedded in the code the shell executes, while when passing it as an argument it's data separate from the code. Commented Apr 27, 2022 at 20:33
  • 1
    @DanielKaplan, exactly, it's just something that goes to $0. It has to be there for the following args to go to to $1, $2, ... Only the ones from 1 up show in $@, and $0 shows in some error messages. Commented Apr 27, 2022 at 21:42
  • 3
    @DanielKaplan, honestly, embedding filenames in the shell code (the sh -c "echo {} ...") is just a bad idea. I can't see it would have any upsides... Commented Apr 27, 2022 at 22:06
1

Even though -exec echo {} avoids shell processing, many versions of echo mangle argument(s) containing backslash or leading hyphen. (And of course it doesn't pipe to wc as you want.)

Instead of executing wc for each pathname, I would use a program that is designed to handle multiple input lines aka records:

find . | awk '{print length}'   # basecase

# if in a multibyte locale and you want bytes not chars
# prefix the awk with LANG=C (or any other singlebyte)

# if pathname (ever) contains newline, and you have GNU find (and awk?)
find . -print0 | awk -vRS='\0' '{print length}'

Or perl -nle 'print length' which defaults to bytes, but I don't see a way to handle the -print0 case to allow newlines.

Improve this answer
3
  • Perl has a -0 option for NUL-separated input. find . -print0 | perl -0ne 'print length, "\n"'. See man perlrun. Perl also supports multi-byte unicode strings in case you want the character length no matter how many bytes each character might take. Simple unicode support can be enable with the -C option. e.g. echo पशुपतिरपि तान्यहानि कृच्छ्राद् | perl -lne 'print length' reports 86 bytes, but echo पशुपतिरपि तान्यहानि कृच्छ्राद् | perl -C -lne 'print length' reports 30 characters. Unicode support can be very complicated - see also man perlunicode Commented Apr 28, 2022 at 6:31
  • oops, that should be perl -0lne, otherwise it will include the NUL byte in the length. Alternatively, you can do calculations in the script - e.g. find . -type f -print0 | perl -0ne 'print length($_) - 1, "\t$_\n" Commented Apr 28, 2022 at 6:42
  • I appreciate you taking the time to write this. I learned things from your answer, but as I said, the command is not my question. This solution is a one-off, and if I was trying to use a different command, this information probably wouldn't apply. I'm looking for something general-purpose; something that'll help me with commands in the future that I haven't even thought of yet. Commented Apr 28, 2022 at 19:12

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.