Questions tagged [floating-point]
The floating-point tag has no summary.
96 questions
2
votes
1
answer
250
views
bc inconsistent trailing zero result as part of a bash script variable
Consider:
heightScaled=$(bc <<< $H*$S/1)
echo $heightScaled
45.0
Why am I getting 45.0 despite the same operations yielding 45 if used outside the script either interactively or piped into ...
- 29
0
votes
0
answers
35
views
How can I apply a function to cells in place in Calc?
I have a large table of numbers using the wrong decimal separator for my locale (dot instead of comma), so LibreOffice Calc does not parse them as numbers, but as strings. I can apply this function to ...
- 141
7
votes
3
answers
447
views
Bash printf float formatting became nonsensical and random
Bash printf floating number formatting (with %f or %g) is suddenly completely wrong, and changing all the time.
An example output:
$ export LC_ALL=C
$ printf '%g\n' 1
1.20739e+3531
$ printf '%g\n' 1
4....
- 751
1
vote
1
answer
81
views
Pi Zero Cross Compilation Floating Point Specification
I am trying to cross compile a go project (syzkaller) to target a Pi Zero so it can emulate usb devices. The problem is that I have been unable to find information regarding how to specify the proper ...
0
votes
0
answers
60
views
Rounding 2.555500000000000104805053 with printf -> 2.555 or 2.556? [duplicate]
I tried
printf "%.3f\n" 2.555500000000000104805053
printf "%.3f\n" 2.555500000000000104805054
the result was
2.555
2.556
same with printf "%.3f\n" 3....
- 167
1
vote
1
answer
346
views
Completed stumped by trying to use a decimal value in an array for Bash
Here is some simple test code.
#!bin/bash
cpm=(0 1 0.094)
lv=1
attack=5
defense=9
stamina=16
echo $((cpm[lv]))
mycpm=$((cpm[lv]))
#mycpm=`echo "0.094" | bc -l`
cq=`echo "$attack*$...
- 73
2
votes
3
answers
2k
views
Replicating floating point arithmetic WITHOUT bc utility
I'm needing to write a script on some embedded Linux systems that don't have bc installed on them and won't be able to have it installed. The script I'm writing is essentially a monitoring script that ...
- 43
0
votes
3
answers
3k
views
Using numbers with decimal points in if statements and operators
I'm trying to write a script that detects the Linux OS and version and through the use of conditions and operators execute different commands depending on the result.
Detect OS
if [ -f /etc/os-release ...
- 67
1
vote
1
answer
2k
views
Invalid arithmetic operator (error token is " 7.16.3 when passing float from read command
I am trying to automate software update with bash script. When I am passing version number e.g 7.16.3 I get following error: ") Syntax error Invalid arithmetic operator (error token is "
I ...
0
votes
1
answer
114
views
My calculation is getting converted to integer by default(KSH) [duplicate]
I am assigning a decimal value to a variable which is an output from an oracle command. when I echo the variable I see the decimal value.
For example, echo $a is showing me .7214 but when I do ...
- 1
0
votes
4
answers
642
views
Floating point comparison in shell
I am comparing the floating point values in shell script based on this reference. Following is the script contents
num1=50.960
num2=6.65E+07
echo "${num1} < ${num2}" | bc
When I ran the ...
- 133
0
votes
2
answers
931
views
Convert a textual representation of a float to actual numeral using command line
I have a double printed as text. Is there a way to pass that source text number to some utility in order to get the actual numeral i.e. the nearest actual representable double value? Either in decimal/...
- 1,499
0
votes
1
answer
386
views
How to shorten decimal output in bash?
Right now I am working on an assignment that involves advanced piping, and one of the questions involves calculating a percent and shortening it to 3 decimal places. I've managed to calculate the ...
- 1
2
votes
1
answer
409
views
POSIX floating comparison
working with a legacy system, here I'm connecting with sqlplus to store the returned value in a variable and its worked fine with me but the problem with compearing with floating values a numeric/...
- 25
1
vote
2
answers
524
views
How many decimal places could awk's rand() give you?
How many decimal places will rand() give?
I assume rand() value can't be any completely arbitrary number from 0 up to excluding 1 and that it's limited to a certain amount of decimal places or ...
- 341