How to diff two files in bash efficiently?

Question

I have two files containing the similar pattern:

cmd1 [cmd2 {xx/xx[7] x/x[0] ...}] cmd3 [cmd4 {xx/x[12] ...}]
cmd5 [cmd6 {x/x[1] xx ...}]

I don't need to consider all the cmds in two files. Only need to sort the string lists' order in braces. Then I sort the two files separately and use comm to output the similarities and differences. I use the same flow to sort the string lists in two files. The flow is below:

matchedBraces=$(grep -o '\{[^}]*\}' $fileA)   #grep all the braces and strings in them

while read perMatch
do
sort_now "$perMatch" $fileA
done <<< "$matchedBraces"

function sort_now {
beforeSort=$(echo "$1" | sed 's?\[?\\[?g' | sed 's?\]?\\]?g')   #in case string has square brackets in it, change [...] to \[...\] for later use
afterSort=$(echo "$1" | grep -o '[^{} ]*' | sort | tr '\n' ' ')   #get all the individual strings in brace, sort them, put them as a string with a trailing white space 
afterSort={$(echo $afterSort)}   #delete the trailing white space,  add the brace back to the string list
afterSort=$(echo "$afterSort" | sed 's?\[?\\[?g' | sed 's?\]?\\]?g')   #in case string has square brackets in it, change [...] to \[...\] for later use
sed -i -f - $2 << eof
s?${beforeSort}?${afterSort}?g
eof   #the variables may be very large, so I have to use sed this way 
sort $2 -o $2
}

I changed the code's order for clarity. It works flawlessly but if a file only contains the pattern cmd1 [cmd2 {xx/xx}] only one string in brace, but there's 50k similar lines, then it's very time-consuming. Even if I put it running with 8cpus 200G mem, it keeps running after hours. Since I know in tcl, to append something to a string, the command append is much more efficient than set. I'm wondering if bash has similar commands or features. Or can someone optimise my code to save time?

Answer 1

#!/bin/bash

# Function to sort string lists within braces
sort_string_lists() {
    local file="$1"
    local matchedBraces=$(grep -o '\{[^}]*\}' "$file")
    
    while read perMatch; do
        sort_now "$perMatch" "$file"
    done <<< "$matchedBraces"
}

# Function to sort the content of a file
sort_now() {
    local match="$1"
    local file="$2"
    
    beforeSort=$(echo "$match" | sed 's?\[?\\[?g' | sed 's?\]?\\]?g')
    afterSort=$(echo "$match" | grep -o '[^{} ]*' | sort | tr '\n' ' ')
    afterSort="{$(echo $afterSort)}"
    afterSort=$(echo "$afterSort" | sed 's?\[?\\[?g' | sed 's?\]?\\]?g')
    
    sed -i -f - "$file" << EOF
s?${beforeSort}?${afterSort}?g
EOF
    
    sort "$file" -o "$file"
}

# Main script starts here
fileA="path/to/fileA.txt"
fileB="path/to/fileB.txt"

# Sort string lists in both files
sort_string_lists "$fileA"
sort_string_lists "$fileB"

# Compare sorted files using comm
comm -23 "$fileA" "$fileB"

This script does the following:

• Defines a function sort_string_lists to extract and sort the string lists in curly braces for a given file.
• Defines a helper function sort_now to perform the actual sorting of the string lists.
• Calls sort_string_lists on both input files to make sure they are sorted according to the criteria you specify.
• Uses comm -23 to compare the sorted files and output the differences (lines only for fileB).

Answer 2

The answer to the question "How to diff two files in bash efficiently?" - use the diff command - a quick google will tell you that.

https://askubuntu.com/questions/515900/how-to-compare-two-files

I'm not sure we should be helping you with your hw assignment.