Search code examples
Is floating-point math broken?...


mathfloating-pointlanguage-agnosticprecisionfloating-accuracy

Read More
Looking for suitable datatype in C to handle two mathematical expressions...


ctypesdoubleprecisionunsigned-long-long-int

Read More
Next higher/lower IEEE double precision number...


floating-pointdoubleprecisionieee-754

Read More
What range of numbers can be represented in 16-, 32-, and 64-bit IEEE-754 systems?...


floating-pointprecisionnumericieee-754

Read More
Does std::to_chars ever really disambiguate using round_to_nearest?...


c++floating-pointlanguage-lawyerprecisionstringification

Read More
How to handle very small numbers (<1e-15) in Python, passed to Scipy/Numpy functions?...


numpyscipyprecisionfloating-accuracympmath

Read More
Why are C++ and Python giving me different answers, when given (I think) the same precision?...


pythonc++precisionmpmath

Read More
Preserving large numbers...


rcsvprecisionnumericr-faq

Read More
Strange output in comparison of float with float literal...


c++cfloating-pointprecision

Read More
PHP microtime() conversion to microseconds...


phpfloating-pointnumbersprecisionmicrotime

Read More
the pow() function isn't accurate...


pythonfloating-pointprecisionpari

Read More
Half-precision floating-point in Java...


javafloating-pointieee-754precision

Read More
Need h‭elp understanding Google Sheets PERCENTIF...


google-sheetsgoogle-sheets-formulaprecision

Read More
Is there an accurate approximation of the acos() function?...


openglglslprecisiontrigonometry

Read More
Inaccuracy replicating Fortran mixed-precision expression in Rust...


rustfloating-pointprecisionfloating-accuracy

Read More
How to properly round-up half float numbers?...


pythonpython-3.xfloating-pointroundingprecision

Read More
How to change `MPFR_MAX_PREC` either via `R` or a re-compile?...


precisiongmpmpfr

Read More
Different result when passing an entry and the same entry which is inside a matrix in Fortran...


fortranprecision

Read More
Can lambdify return an array with a dtype np.float128?...


pythonnumpyfloating-pointsympyprecision

Read More
integer exact computation with logs...


floating-pointprecisionnumerical-methodsfloating-accuracy

Read More
PHP7.1 json_encode() Float Issue...


phpjsonprecisionphp-7.1

Read More
Math.pow(64, 1/3) returns inexact 4...


javascriptmathprecision

Read More
How to 'cout' the correct number of decimal places of a double value?...


c++doubletostringprecision

Read More
Has anyone encounter this this stripping artifact during "RayTracing in one Weekend"?...


randomprecisionraytracingcompute-shader

Read More
Regex for Oracle NUMBER(10,8) field...


regexoracle-databaseprecisionfixed-point

Read More
Big numbers in c++ issue in cosine similarity...


c++precisionlong-integer

Read More
How are FLT_DIG, DBL_DIG, and LDBL_DIG determined in C...


cfloating-pointprecisionfloating-accuracy

Read More
Which single-precision floating point numbers need 9 significant decimal digits for unambiguous deci...


floating-pointprecision

Read More
Is the Java float to double cast lossy?...


javacastingfloating-pointprecision

Read More
Integer stored in 0 bytes...


sqlitememory-managementprecision

Read More
BackNext