Skip to main content
Software Engineering

Questions tagged [floating-point]

Ask Question

The tag has no summary.

84 questions
Newest Active Bountied Unanswered
2 votes
2 answers
382 views

There is already a prior question dealing with why certain bit-widths were chosen (although I do find it somewhat insufficient, but that's another topic), but what strikes me as unusual is how the ...
petroleus's user avatar
  • 132
6 votes
6 answers
669 views

Several random number generators I've used return numbers between 0 and 1, such as JavaScript's Math.random() It's also notoriously true that floating point numbers have an issue with storing specific ...
David Starkey's user avatar
  • 201
0 votes
6 answers
1k views

I'm writing a math library and I've encountered trouble with floating point imprecision. I want to know if there are any standard workarounds that are used often. I have doubts about my current method ...
Scene's user avatar
  • 149
-4 votes
2 answers
147 views

I’m starting at numerical methods and I am introducing to the machine epsilon and succesor. I have this question, which is the succesor of .13579x10^2 at a floating point system of (10,5,-2,6). My ...
RES's user avatar
  • 3
7 votes
2 answers
697 views

I've been reading about the IEEE 754 standard, but can't understand how or why they chose those exact number of bits for the exponent and mantissa. For example, why 8 bits for the exponent and not 7 ...
hackm's user avatar
  • 73
-4 votes
1 answer
196 views

floating point representations## exponent == 7 bits ,mantissa == 8 bits 2044    = 0 1001001 11111111 2045.5 = 0 1001001 11111111 why is this happening?? what is the error? Is anything wrong with the ...
Senu Nawamina's user avatar
  • 1
-1 votes
3 answers
784 views

I was looking into a datasheet of a Real Time Clock (RTC) with a temperature sensor MAX31328. (You can find the datasheet easily). I'd like to convert the raw binary value into decimal format. I have ...
Nuz's user avatar
  • 51
7 votes
2 answers
2k views

My idea is that, if an unsigned 32-bit integer is used for representing angles in a way that 0 is 0°, 231 is 180° and 232−1 is just under 360°, all possible values are used and evenly distributed ...
matj1's user avatar
  • 187
2 votes
4 answers
323 views

Having a bit of trouble on deciding the best way to store crypto values which are usually in decimals. Do I go with Double or Float? Or is there an even better type that Swift offers? My initial gut ...
Tank12's user avatar
  • 139
7 votes
6 answers
3k views

My understanding is that it's rarely a good idea to compare two floating point numbers due to inherent inaccuracies. However, my understanding of those inaccuracies are that they are deterministic and ...
user avatar
-4 votes
1 answer
233 views

Apart from the memory size of both data types, I want to know any other differences and also want to know when we use float and when we use double?
lokeshjoshi's user avatar
  • 1
2 votes
2 answers
1k views

I am trying to write an algorithm to accurately calculate exponents (antilogs) for a variable precision floating point library I am working on. The base is not relevant since I can convert between ...
user264480's user avatar
  • 81
-1 votes
5 answers
3k views

What Every Computer Scientist Should Know About Floating-point Arithmetic is widely considered absolutely mandatory reading for every programmer. Why is this the case? What aspects of the article make ...
J. Mini's user avatar
  • 1,015
6 votes
4 answers
4k views

I am writing numerical calculation software using .NET C#, which needs to be blazingly fast. There is a lot of fractional math. So using decimal type is pretty much out of the question, given its poor ...
Brie's user avatar
  • 288
6 votes
2 answers
6k views

Firstly i am aware that different hardware architectures handle floats differently, so I can't guarantee the same result on different machines even with the same inputs. But in my case i do not have a ...
WDUK's user avatar
  • 2,092

15 30 50 per page
1
2 3 4 5 6