Subscribe to this thread
Home - Cutting Edge / All posts - RoundDecs problem?
RAR61 post(s)
#17-Sep-20 21:09

It seems like I am having issues with RoundDecs(value,2) function: it shows say 1.67 as 1.6999999xxx quite few records.

How would we get just 2 decimals consistently please?

I am using the latest build.

Thank you

tjhb

9,452 post(s)
online
#17-Sep-20 21:51

That is normally a binary representation issue. (Most decimal fractions can’t be expressed exactly in binary. Only those which can be expressed as a sum of fractions each having denominator equal to a power of 2.)

You will need to use a string to get exact representation for all floating-point numbers.

I can’t test just now, but I expect it is enough to CAST([field] AS NVARCHAR). You can do this as a new computed field of type NVARCHAR.

You may to introduce explicit rounding on the way; hopefully not.

RAR61 post(s)
#17-Sep-20 21:59

Yes, that cast would solve it but the field will be a nvarchar though.

Thank you so much Tim!

RAR61 post(s)
#17-Sep-20 22:09

Tim,

I was wrong: casting is actually rounding it properly in a float64 field.

Thank you so much: it solved my problem!!!

RAR61 post(s)
#17-Sep-20 22:17

Very strange: i was testing it before and it worked. Now it complains about a type mismatch in the real data processing.

I don't understand.

Thank you again!

tjhb

9,452 post(s)
online
#21-Sep-20 22:21

I have only just tested this, and using the value 1.67 (and some others) I don't see a problem.

It seems to me that Manifold 9.0.172.x treats even "tricky" floating-point values correctly. (That is, it is applying some cosmetic rounding behind the scenes, to show decimal values as we would expect.)

So two questions: what build are you using, and what exact values are you using?

Can you supply some sample data so that we can work on this?

RAR61 post(s)
#22-Sep-20 00:39

Tim,

I was trying to create a sample data that you can test but as I mentioned before I am confused as sometimes it does it properly, sometimes not - and that for the same samples. Would it be something in my computer or something happening during the calculation? I truly do not now.

Thank you for your help though.

tjhb

9,452 post(s)
online
#22-Sep-20 01:11

That is odd! I should have read you more literally before sorry.

Maybe if and when you see it again, take a screen picture, so there is at least some context or reminder. We could replicate as much as possible, maybe figure out how to automate some testing.

E.g. whether in a query, the result of transform, whether multiple CPU threads or GPGPU are being used, anything is useful.

RAR61 post(s)
#22-Sep-20 01:38

I definitely will when I see that again; I really want to know why is it happening.

Thank you Tim!

tjhb

9,452 post(s)
online
#22-Sep-20 03:13

You might want to have a look at this thread, to see if it rings any bells. It might not.

RAR61 post(s)
#22-Sep-20 12:35

Tim,

Thank you so much for your assistance: this link is really helpful.

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.