374 questions
Best practices
0
votes
9
replies
138
views
Reducing significant digits and eliminating scientific notation when float/double approaches zero
Surely, a number of ways are at a developer's disposal to reduce a rounding error of 1.78814e-07 variable assignment to 0.000000. What is best practice?
My best solution is to include a method as such:...
- 11
0
votes
1
answer
120
views
Why do the Centroid function and centroid method return (slightly) different results
I have a GeoDjango (v5.2) model ("Location") with a MultiPolygonField ("coordinates") and use PostGIS (v17-3.5) as database backend. When I annotate a queryset with the multi-...
- 76
0
votes
0
answers
72
views
Is there a known issue with Spark version 3 behaving differently and inconsistent to Spark version 2 when rounding decimal points?
My project is having a problem with decimal rounding errors in Spark v3 compared to Spark v2, where rounding to second decimal place works fine in Spark v2 but Spark v3 rounds the values up or down (...
0
votes
2
answers
69
views
Why isn't the signif function working for Banker's even rounding for 0.6745? [duplicate]
For this example, I am working with the following data:
Z_Value <- c(-0.6745, 0.6745)
I'm using the following function and want to use 3 sig figs. (If I have results that are 999 or higher, I want ...
- 59
0
votes
2
answers
71
views
Round to n or less decimal places [duplicate]
I'm trying to round the output of some calculations to 3 or less decimal places, so they can stored compactly in a comma-delimited text file. That is 2 should be stored as 2 not 2.000, 3.14 should be ...
- 1,132
0
votes
1
answer
78
views
How do I fix the floating point problem in this case? [duplicate]
I'm trying to make a code for a simple calculator for a programming problem, and I can't figure out how the rounding works in this case.
1 + 0 + 0.0005 returns 1.000 but 2 + 1 + 0.0005 returns 3.001.
...
- 1
1
vote
2
answers
193
views
Example of the utility of the `G9` format specifier in C# for round-tripping floating-point numbers?
I'm trying to understand the practical utility of the G9 format specifier in C# when dealing with round-tripping floating-point numbers.
The book C#12 in a nutshell mentions that G9 is helpful to ...
3
votes
2
answers
228
views
Truncating floats to int on x86 vs x86_64 has different rounding error
I stumbled upon a failing unit-test, that involves converting doubles to integers.
The actual number that is converted is 1.234 * 1000., and the code basically boils down to:
#include <iostream>
...
- 32.4k
2
votes
1
answer
129
views
Collision detection for concave polygons with SAT, round/off error optimization
Basically I want to create my own tools for developing a 2D top-down game and when it comes to collision detection I'm trying to implement the SAT method, but somehow my calculations seem to be way ...
1
vote
1
answer
378
views
HLSL: Force order of floating point operations to prevent rounding error
When writing shaders in HLSL, sometimes you can run into very large floating point numbers, and with large floating point numbers, they struggle to maintain precision. When I wrote the blow code, I ...
- 1,070
2
votes
1
answer
360
views
Using MTLPixelFormat.rgba16Float results in random round-off errors
I'm using a Metal compute pipeline to render into a CAMetalLayer's drawable and keep running into problems with non-integer pixel formats resulting in a slightly noisy image, due to some apparent ...
- 152
0
votes
2
answers
93
views
why does the default round function in python fix double rounding errors?
There's something wrong with my understanding of either the round function in python or how doubles are represented in python. I find a very perplexing behavior when rounding numbers that have a ...
- 101
0
votes
3
answers
194
views
MariaDB inconsistent comparison results between DOUBLE(10,2) and string
I'm searching a table for an exact match on an amount column DOUBLE(10,2). Due to the database driver I'm using, I have to bind my amount as a string parameter. In general this works well, hoever this ...
- 29
-1
votes
1
answer
79
views
Can't use double, can't use int, what do I use to stop the rounding? (Java) [duplicate]
I am incredibly new to java and this is the description for a lab I currently have:
Statistics are often calculated with varying amounts of input data. Write a program that takes any number of non-...
0
votes
1
answer
78
views
R: Allocating N discrete elements between population cohorts by probability
I want to model demographic changes within a population, using historical data going from 1986 until the present. My data covers a population divided into age cohorts of a single year: 0,1,2 ... 106+.
...
- 59