All Questions
Tagged with performance shell-script
37 questions
0
votes
0
answers
55
views
Is there a way to make window either spawning or changing from tiled to floating take the earlier size and place?
I know that I can tell bspwm to spawn windows in specified desktop or monitor in tiled mode or in certain shape and position via floating mode. But I am looking for the window to remember its previous ...
- 11
2
votes
1
answer
632
views
whisper.cpp takes a long time to start. Can this be avoided?
I am using https://unix.stackexchange.com/users/560724/georgi-gerganov 's whisper.cpp to transcribe recorded interviews. Many of these are short, a minuter or two. The problem is that whisper.cpp ...
- 2,047
8
votes
5
answers
8k
views
Most efficient way of changing 1 line in a file
I want to change the first line of hundreds of files recursively in the most efficient way possible. An example of what I want to do is to change #!/bin/bash to #!/bin/sh, so I came up with this ...
- 123
1
vote
2
answers
952
views
How to read file every Nth character the fastest way possible?
I already know a couple of ways to read a file every Nth characters in bash:
LC_ALL=C
while read -n100 character; do
echo "$character"
done < <(cat "$@" | tr -d '\n')
...
- 2,462
7
votes
2
answers
2k
views
Efficient way to do calculations in bash
I'm trying to calculate the geometric mean of a file full of numbers (1 column).
The basic formula for geometric mean is the average the natural log (or log) of all the values and then raise e (or ...
- 83
0
votes
1
answer
110
views
No response of egrep command on large folder
I need to get some files that doesn't contain some string on a large folder ≃ 10M files for 22 GO of data.
I try this command on local (macOS) :
egrep -r -L -Z 'string1|string2' * | wc -l
this work ...
- 103
1
vote
2
answers
1k
views
Alternative of too much slow gzip -d command
I'm using an production server for loading large data set to Hadoop to access from Hive table.
We are loading subscribers web browsing data of Telecom Sector. We've large number of .csv.gz file (File ...
- 259
30
votes
11
answers
9k
views
Why doesn't shell automatically fix "useless use of cat"? [closed]
Many people use oneliners and scripts containing code along the lines
cat "$MYFILE" | command1 | command2 > "$OUTPUT"
The first cat is often called "useless use of cat" because technically it ...
- 4,359
1
vote
2
answers
2k
views
How to search for large directory containing thousands of files?
Under the folder
/grid/sdh/hadoop/yarn/local/usercache/hdfs/appcache
we have more than 100 recursive folders.
One of the folders contain thousands of files.
Is it possible to identify this folder?
...
- 13.9k
3
votes
1
answer
2k
views
filtering a large file with a large filter
I want to extract all lines of $file1 that start with a string stored in $file2.
$file1 is 4 GB large with about 20 million lines, $file2 has 2 million lines, is about 140 MB large and contains two ...
- 354
7
votes
1
answer
2k
views
efficiently grep an interval of a sorted file
My file has millions of lines, resides in memory /dev/shm/tmp.file, is accessed by multiple threads, looks like this
831092,25a1bd66f2eec71aa2f0a8bb3d,/path/to/a/file
4324,8d83c29e4d8c71bd66f1bd66fs,/...
- 354
27
votes
9
answers
4k
views
Is piping, shifting, or parameter expansion more efficient?
I'm trying to find the most efficient way to iterate through certain values that are a consistent number of values away from each other in a space separated list of words(I don't want to use an array)....
- 458
0
votes
2
answers
738
views
How to remove trailing CR quickly?
This is my source file, it dumps a database into a JSON and worked great with Debian wheezy:
#!/bin/sh
echo "Status: 200"
echo
echo {
printf ".mode csv\nselect * from cfg;" | sqlite3 /home/ftproot/...
- 3,563
2
votes
1
answer
207
views
Zsh Performance: Should I create scripts or just add functions to a file I source in .zshrc?
I use a lot of little zsh functions. I have been adding them to a single functions.zsh file which I source in .zshrc.
If they are created as scripts, or autoloading is used, will that improve speed?
...
- 1,684
1
vote
1
answer
1k
views
Possible to speed this while read bash script up?
I'm still VERY new to the whole linux bash scripting and I've come up with this little bit of code, unfortunately, its execution time is kinda long for a small file (193 KiB)
real 0m7.234s
user ...
- 13