Skip to content

Missing Data, Rounding, Clipping and isclose

Real data is messy. Sensors miss readings, forms are left blank, and calculations give long decimals. This lesson covers the tools to deal with that:

  • Missing values: np.nan, np.isnan, and how to remove or fill them.
  • Rounding: round, floor, ceil, trunc.
  • Clipping: keeping values inside a range with np.clip.
  • Comparing floats safely with np.isclose and np.allclose.

How to run the examples

Create a new file named missing_rounding_clipping.py in your NumpyTutorials folder (see the setup in lesson 01). Any name works; this one just matches the topic. Start the file with these lines:

import numpy as np

Then copy the examples of this page into the same file, one after another, in the order shown, and run it with python missing_rounding_clipping.py (with your virtual environment active). Some examples use variables created in earlier ones, so keep everything in the one file. Each Output box shows what that example prints.

1. What is np.nan?

np.nan means "Not a Number". NumPy uses it to mark a missing value.

print("type        :", type(np.nan))
print("nan == nan  :", np.nan == np.nan, "<- nan is never equal to anything!")
print("nan + 5     :", np.nan + 5)
print("isnan(nan)  :", np.isnan(np.nan))

readings = np.array([12.5, np.nan, 14.0, np.nan, 15.5])
print("readings          :", readings)
print("readings == nan   :", readings == np.nan, "<- wrong way")
print("np.isnan(readings):", np.isnan(readings), "<- right way")
Output
type        : <class 'float'>
nan == nan  : False <- nan is never equal to anything!
nan + 5     : nan
isnan(nan)  : True
readings          : [12.5  nan 14.   nan 15.5]
readings == nan   : [False False False False False] <- wrong way
np.isnan(readings): [False  True False  True False] <- right way
  • nan is a float.
  • Any maths with nan gives nan: nan + 5 is nan.
  • nan is not equal to anything, not even itself. So readings == np.nan is always all False. This is a classic bug.
  • The only correct test is np.isnan().

2. nan only fits in float arrays

counts = np.array([5, 7, 9])
try:
    counts[1] = np.nan
except ValueError as error:
    print("Error:", error)

counts = counts.astype(float)
counts[1] = np.nan
print("after astype(float):", counts)
Output
Error: cannot convert float NaN to integer
after astype(float): [ 5. nan  9.]

Integer arrays have no way to store "missing". Convert the array to float first with astype(float), then put nan in. This is also why a column of whole numbers with a gap becomes float when you load it (lesson 15).

3. Find, count, remove

print("how many missing :", np.isnan(readings).sum())
print("where            :", np.where(np.isnan(readings))[0])
print("without nan      :", readings[~np.isnan(readings)])

table = np.array([[1.0, 2.0, 3.0],
                  [4.0, np.nan, 6.0],
                  [7.0, 8.0, 9.0]])
complete_rows = ~np.isnan(table).any(axis=1)
print("rows with no nan :\n", table[complete_rows])
Output
how many missing : 2
where            : [1 3]
without nan      : [12.5 14.  15.5]
rows with no nan :
 [[1. 2. 3.]
 [7. 8. 9.]]
  • np.isnan(arr).sum() counts the missing values (True = 1).
  • np.where(np.isnan(arr))[0] gives their positions.
  • arr[~np.isnan(arr)] keeps only the real values.
  • For a table, np.isnan(table).any(axis=1) is True for rows with at least one gap. With ~, you keep only complete rows.

4. Fill missing values

Instead of removing rows, you can fill the gaps:

filled_zero = np.nan_to_num(readings, nan=0.0)
print("nan -> 0          :", filled_zero)

mean = np.nanmean(readings)
filled_mean = np.where(np.isnan(readings), mean, readings)
print("nan -> mean", round(mean, 2), ":", filled_mean.round(2))

column_means = np.nanmean(table, axis=0)
rows, cols = np.where(np.isnan(table))
table_filled = table.copy()
table_filled[rows, cols] = column_means[cols]
print("table with column mean filled in:\n", table_filled)
Output
nan -> 0          : [12.5  0.  14.   0.  15.5]
nan -> mean 14.0 : [12.5 14.  14.  14.  15.5]
table with column mean filled in:
 [[1. 2. 3.]
 [4. 5. 6.]
 [7. 8. 9.]]
  • np.nan_to_num(arr, nan=0.0) replaces nan with a fixed value.
  • np.where(np.isnan(arr), mean, arr) fills the gaps with the average of the real values (np.nanmean, lesson 11).
  • For a table, calculate the mean of each column, find the row and column of each gap with np.where, and put that column's mean there with fancy indexing.

5. inf, isinf, isfinite

values = np.array([1.0, np.inf, -np.inf, np.nan, 5.0])
print("isinf    :", np.isinf(values))
print("isfinite :", np.isfinite(values), "(False for inf AND nan)")
print("only real numbers:", values[np.isfinite(values)])
Output
isinf    : [False  True  True False False]
isfinite : [ True False False False  True] (False for inf AND nan)
only real numbers: [1. 5.]

inf (infinity) appears after dividing by zero (lesson 09). np.isinf finds it, and np.isfinite is True only for normal numbers. That makes arr[np.isfinite(arr)] a one-step way to drop both nan and inf.

6. round, floor, ceil, trunc

prices = np.array([12.345, 7.891, -3.456, 99.999])
print("round(2)  :", np.round(prices, 2))
print("round(0)  :", np.round(prices))
print("round(-1) :", np.round(np.array([123, 456, 789]), -1), "(to tens)")
print("floor     :", np.floor(prices), "(down)")
print("ceil      :", np.ceil(prices), "(up)")
print("trunc     :", np.trunc(prices), "(cut decimals, towards 0)")

halves = np.array([0.5, 1.5, 2.5, 3.5])
print("round of .5 values:", np.round(halves), "<- 'round half to even' (banker's rounding)")
print("always round .5 up:", np.floor(halves + 0.5))
Output
round(2)  : [ 12.34   7.89  -3.46 100.  ]
round(0)  : [ 12.   8.  -3. 100.]
round(-1) : [120 460 790] (to tens)
floor     : [12.  7. -4. 99.] (down)
ceil      : [ 13.   8.  -3. 100.] (up)
trunc     : [12.  7. -3. 99.] (cut decimals, towards 0)
round of .5 values: [0. 2. 2. 4.] <- 'round half to even' (banker's rounding)
always round .5 up: [1. 2. 3. 4.]
FunctionDoes-3.456 becomes
np.round(x, 2)round to 2 decimals-3.46
np.floor(x)round down-4
np.ceil(x)round up-3
np.trunc(x)cut the decimals (towards 0)-3
  • A negative number of decimals rounds to tens, hundreds and so on: np.round(456, -1) gives 460.
  • The results are still floats (12.). Add .astype(int) if you need integers.

Surprise: np.round([0.5, 1.5, 2.5, 3.5]) gives [0, 2, 2, 4], not [1, 2, 3, 4]. NumPy (and Python's round) uses "round half to even", also called banker's rounding: a value exactly halfway goes to the nearest even number. Over many values this avoids always pushing totals upwards. If you need the school rule "0.5 always goes up", use np.floor(x + 0.5).

7. np.clip

np.clip(arr, low, high) keeps every value inside a range: values below low become low, values above high become high.

marks = np.array([-5, 45, 88, 105, 72])
print("clip 0..100      :", np.clip(marks, 0, 100))
print("at least 35      :", np.clip(marks, 35, None))
print("same as maximum/minimum:", np.minimum(np.maximum(marks, 0), 100))

pixels = np.array([100, 200, 250], dtype=np.int16)
print("brighter, safe   :", np.clip(pixels + 30, 0, 255))     # used in lesson 17
Output
clip 0..100      : [  0  45  88 100  72]
at least 35      : [ 35  45  88 105  72]
same as maximum/minimum: [  0  45  88 100  72]
brighter, safe   : [130 230 255]
  • Marks of -5 and 105 (typing mistakes) become 0 and 100.
  • None means "no limit on this side".
  • It is the same as np.minimum(np.maximum(arr, low), high), but shorter and clearer.
  • Clipping is essential for images: pixel values must stay between 0 and 255 (lesson 17).

8. isclose and allclose

Computers store decimals in binary, so many simple decimals, like 0.1, cannot be stored exactly. Tiny errors appear, and == fails:

print("0.1 + 0.2        :", 0.1 + 0.2)
print("0.1 + 0.2 == 0.3 :", 0.1 + 0.2 == 0.3)
print("np.isclose       :", np.isclose(0.1 + 0.2, 0.3))

a = np.array([1.0, 2.0, 3.0])
b = np.array([1.0000001, 2.0, 3.01])
print("isclose(a, b)    :", np.isclose(a, b))
print("with atol=0.05   :", np.isclose(a, b, atol=0.05))
print("allclose(a, b)   :", np.allclose(a, b))
print("isclose nan, nan :", np.isclose(np.nan, np.nan), "/ with equal_nan=True:",
      np.isclose(np.nan, np.nan, equal_nan=True))
Output
0.1 + 0.2        : 0.30000000000000004
0.1 + 0.2 == 0.3 : False
np.isclose       : True
isclose(a, b)    : [ True  True False]
with atol=0.05   : [ True  True  True]
allclose(a, b)   : False
isclose nan, nan : False / with equal_nan=True: True
  • 0.1 + 0.2 is really 0.30000000000000004, so == 0.3 is False.
  • np.isclose(a, b) is True when the numbers are equal within a tiny tolerance. Use it whenever you compare floats.
  • atol sets your own tolerance: with 0.05, 3.0 and 3.01 count as close.
  • np.allclose gives one True/False for whole arrays.
  • nan is not close to nan unless you pass equal_nan=True.

9. Real example: temperature sensor

raw = np.array([24.6, 25.1, np.nan, 250.0, 24.9, -40.0, np.nan, 25.4])

clean = raw.copy()
clean[(clean < -10) | (clean > 60)] = np.nan        # impossible values -> missing
print("impossible -> nan :", clean)
clean = np.where(np.isnan(clean), np.nanmedian(clean), clean)
print("filled with median:", clean)
print("rounded to 0.5    :", np.round(clean * 2) / 2)
print("average           :", clean.mean().round(2))
Output
impossible -> nan : [24.6 25.1  nan  nan 24.9  nan  nan 25.4]
filled with median: [24.6 25.1 25.  25.  24.9 25.  25.  25.4]
rounded to 0.5    : [24.5 25.  25.  25.  25.  25.  25.  25.5]
average           : 25.0
  1. Readings of 250 and -40 °C are impossible indoors, so a mask turns them into nan.
  2. All gaps are filled with the median of the good readings. The median is not disturbed by extreme values.
  3. np.round(x * 2) / 2 rounds to the nearest 0.5.

Exercises

Try these exercises on your own before moving on:

  1. Count the missing values in np.array([1, np.nan, 3, np.nan, np.nan]).
  2. Fill the missing values of a 2D array with the mean of each ROW.
  3. Round np.array([2.5, 3.5, 4.5]) and explain the result.
  4. Keep all percentages in np.array([-3, 50, 120]) between 0 and 100.
  5. Check if 0.1 * 3 equals 0.3 with == and with np.isclose.

Summary

  • np.nan marks missing values; it only fits in float arrays.
  • Never use == np.nan; always use np.isnan().
  • Remove gaps with arr[~np.isnan(arr)] or fill them with np.where / np.nan_to_num.
  • round, floor, ceil and trunc round in different ways; .5 rounds to even.
  • np.clip keeps values inside a range.
  • Compare floats with np.isclose/np.allclose, not ==.

Next: Lesson 15: File I/O and Structured Arrays.