Boolean Masking, Fancy Indexing and np.where

Slicing (lesson 04) picks elements by position. Very often you want to pick elements by a condition instead ("all marks below 35", "orders from Pune above 1000"), or by a list of positions in any order. This lesson covers those three tools:

Boolean masking: marks[marks >= 35]
Fancy indexing: marks[[0, 3, 5]]
np.where: np.where(marks >= 35, "Pass", "Fail")

How to run the examples

Create a new file named boolean_fancy_where.py in your NumpyTutorials folder (see the setup in lesson 01). Any name works; this one just matches the topic. Start the file with these lines:

import numpy as np

Then copy the examples of this page into the same file, one after another, in the order shown, and run it with python boolean_fancy_where.py (with your virtual environment active). Some examples use variables created in earlier ones, so keep everything in the one file. Each Output box shows what that example prints.

Most examples use these two arrays, so add them right after the import line:

names = np.array(["Asha", "Ravi", "Neha", "Kiran", "Meena", "Arjun"])
marks = np.array([82, 34, 67, 91, 28, 55])

1. A comparison gives a boolean mask

Comparing an array with a value gives an array of True/False, one per element. This is called a mask. Putting a mask inside the brackets keeps only the elements where it is True.

passed = marks >= 35
print("marks          :", marks)
print("marks >= 35    :", passed)
print("passed marks   :", marks[passed])       # keep only True positions
print("passed names   :", names[passed])       # the same mask works on another array
print("number passed  :", passed.sum())        # True counts as 1
print("failed names   :", names[~passed])      # ~ means NOT

Output

marks          : [82 34 67 91 28 55]
marks >= 35    : [ True False  True  True False  True]
passed marks   : [82 67 91 55]
passed names   : ['Asha' 'Neha' 'Kiran' 'Arjun']
number passed  : 4
failed names   : ['Ravi' 'Meena']

marks >= 35 gives [True False True True False True].
marks[passed] keeps the 4 values at the True positions.
The same mask can filter another array of the same length: names[passed] gives the names of the students who passed. This is how you work with related columns of data.
passed.sum() counts the True values (True = 1, False = 0).
~ turns True into False and the other way round.

2. Combining conditions: &, | and ~

# Always put each condition in ( ), because & and | run before > and <
print("60 to 90       :", marks[(marks >= 60) & (marks <= 90)])
print("below 35 or 90+:", marks[(marks < 35) | (marks >= 90)])
print("not 'Ravi'     :", names[~(names == "Ravi")])

try:
    marks[marks >= 60 and marks <= 90]        # 'and' does not work on arrays
except ValueError as error:
    print("Error with 'and':", error)

Output

60 to 90       : [82 67]
below 35 or 90+: [34 91 28]
not 'Ravi'     : ['Asha' 'Neha' 'Kiran' 'Meena' 'Arjun']
Error with 'and': The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Python	NumPy	Meaning
`and`	`&`	both conditions True
`or`	`\|`	at least one True
`not`	`~`	the opposite

Two rules beginners always trip over: use & | ~, not and or not (they raise the "truth value is ambiguous" error shown above), and put each condition in brackets: (marks >= 60) & (marks <= 90).

3. Changing values with a mask

grace = marks.copy()
grace[grace < 35] = 35                        # everyone below 35 gets 35
print("after grace    :", grace)

bonus = marks.copy()
bonus[bonus >= 80] += 5                       # +5 only for 80 and above
print("after bonus    :", bonus)

Output

after grace    : [82 35 67 91 35 55]
after bonus    : [87 34 67 96 28 55]

A mask on the left of = changes only the matching elements. grace[grace < 35] = 35 raises every failing mark to 35, and bonus[bonus >= 80] += 5 adds 5 only to the high marks. We use .copy() so the original marks stays the same for the next examples.

4. Masks on a 2D array

sales = np.array([[120, 80, 150],
                  [60, 200, 90],
                  [170, 40, 110]])

print("sales > 100 :\n", sales > 100)
print("values > 100 (1D result):", sales[sales > 100])

# Keep whole ROWS where column 0 is above 100
print("rows where column 0 > 100:\n", sales[sales[:, 0] > 100])

low = sales.copy()
low[low < 100] = 0
print("values below 100 set to 0:\n", low)

Output

sales > 100 :
 [[ True False  True]
 [False  True False]
 [ True False  True]]
values > 100 (1D result): [120 150 200 170 110]
rows where column 0 > 100:
 [[120  80 150]
 [170  40 110]]
values below 100 set to 0:
 [[120   0 150]
 [  0 200   0]
 [170   0 110]]

A mask on a 2D array returns a 1D array of the matching values, because the matching values do not form a rectangle any more.
To keep whole rows, build a 1D mask from one column: sales[:, 0] > 100 is True/False per row, so sales[mask] returns complete rows.
Assigning through a mask works in 2D too.

5. Fancy indexing: a list of indexes

Instead of a slice, you can pass a list (or array) of indexes. NumPy returns the elements at exactly those positions, in exactly that order.

print("marks[[0, 3, 5]]   :", marks[[0, 3, 5]])
print("any order [5, 0]   :", marks[[5, 0]])
print("repeats [1, 1, 1]  :", marks[[1, 1, 1]])
print("names of top 3     :", names[[3, 0, 2]])

table = np.arange(1, 13).reshape(3, 4)
print("table:\n", table)
print("rows 2 and 0       :\n", table[[2, 0]])
print("columns 3 and 1    :\n", table[:, [3, 1]])
print("pairs (0,1),(2,3)  :", table[[0, 2], [1, 3]])          # table[0,1] and table[2,3]
print("rows [0,2] x cols [1,3] with np.ix_:\n", table[np.ix_([0, 2], [1, 3])])

Output

marks[[0, 3, 5]]   : [82 91 55]
any order [5, 0]   : [55 82]
repeats [1, 1, 1]  : [34 34 34]
names of top 3     : ['Kiran' 'Asha' 'Neha']
table:
 [[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]
rows 2 and 0       :
 [[ 9 10 11 12]
 [ 1  2  3  4]]
columns 3 and 1    :
 [[ 4  2]
 [ 8  6]
 [12 10]]
pairs (0,1),(2,3)  : [ 2 12]
rows [0,2] x cols [1,3] with np.ix_:
 [[ 2  4]
 [10 12]]

Any order is allowed, and an index can repeat.
On a 2D array, table[[2, 0]] picks whole rows, and table[:, [3, 1]] picks whole columns.
table[[0, 2], [1, 3]] pairs the lists up: it gives table[0, 1] and table[2, 3], so 2 values, not a 2x2 block.
To get the block of rows 0 and 2 and columns 1 and 3, use np.ix_.

6. Changing values with fancy indexing

stock = np.array([5, 0, 12, 0, 7])
empty_items = np.nonzero(stock == 0)[0]      # indexes where a condition is True
print("empty items   :", empty_items)

stock[empty_items] = 20                      # restock exactly those items
print("after restock :", stock)

stock[[0, 0, 4]] += 1                        # a repeated index is changed only ONCE
print("after [0, 0, 4] += 1:", stock)

Output

empty items   : [1 3]
after restock : [ 5 20 12 20  7]
after [0, 0, 4] += 1: [ 6 20 12 20  8]

np.nonzero(condition) gives the indexes where a condition is True. Here: the empty items are at 1 and 3.
We then use those indexes with fancy indexing to restock exactly those items.
Careful with repeats: stock[[0, 0, 4]] += 1 adds 1 to item 0 only once, not twice. (If you really need repeats counted, use np.add.at(stock, [0, 0, 4], 1).)

7. np.where(condition, value_if_true, value_if_false)

np.where is the vectorized "if / else": it builds a new array, taking the first value where the condition is True and the second where it is False.

result = np.where(marks >= 35, "Pass", "Fail")
print("result        :", result)

capped = np.where(marks > 90, 90, marks)          # keep marks, but at most 90
print("capped at 90  :", capped)

grade = np.where(marks >= 75, "A", np.where(marks >= 50, "B", "C"))   # where inside where
print("grade         :", grade)

# With only the condition, np.where gives the INDEXES where it is True
print("indexes of failures:", np.where(marks < 35)[0])

Output

result        : ['Pass' 'Fail' 'Pass' 'Pass' 'Fail' 'Pass']
capped at 90  : [82 34 67 90 28 55]
grade         : ['A' 'C' 'B' 'A' 'C' 'B']
indexes of failures: [1 4]

The values can be single values ("Pass"/"Fail") or whole arrays (marks).
For three choices, put a where inside a where.
With only the condition, np.where(marks < 35) gives the indexes where it is True (the same as np.nonzero). It returns a tuple, so we take [0].

8. np.select: many conditions

For more than two or three choices, nested where calls become hard to read. np.select takes a list of conditions and a list of results; the first condition that is True wins.

conditions = [marks >= 90, marks >= 75, marks >= 50, marks >= 35]
grades = ["A+", "A", "B", "C"]
print("grade:", np.select(conditions, grades, default="Fail"))

Output

grade: ['A' 'Fail' 'B' 'A+' 'Fail' 'B']

Because 91 matches marks >= 90 first, it gets "A+", even though it also matches the later conditions. Values that match nothing get the default.

9. Real example: orders

order_ids = np.array([101, 102, 103, 104, 105, 106])
amounts = np.array([450, 1200, 3200, 80, 950, 5100])
cities = np.array(["Pune", "Mumbai", "Pune", "Delhi", "Mumbai", "Pune"])

big_pune = (cities == "Pune") & (amounts > 1000)
print("Big Pune orders :", order_ids[big_pune])
print("Delivery charge :", np.where(amounts >= 500, 0, 40))
print("Top 2 orders    :", order_ids[np.argsort(amounts)[-2:]])     # argsort: lesson 12
print("Mumbai total    :", amounts[cities == "Mumbai"].sum())

Output

Big Pune orders : [103 106]
Delivery charge : [40  0  0 40  0  0]
Top 2 orders    : [103 106]
Mumbai total    : 2150

Two conditions on two different arrays: Pune orders and amounts above 1000.
np.where sets a delivery charge of 0 for orders of 500 or more.
np.argsort (lesson 12) gives the order of the amounts, and fancy indexing picks the ids of the top 2.

Boolean masks and fancy indexing always return a copy, while slices return a view. The next lesson explains why that matters.

Exercises

Try these exercises on your own before moving on:

From np.arange(1, 31) get all numbers divisible by 3 but not by 2.
Replace every negative value in np.array([4, -2, 7, -9, 0]) with 0 (once with a mask, once with np.where).
From the sales table get the values of the diagonal with fancy indexing.
Label amounts as "Low" (<500), "Medium" (<2000) or "High" with np.select.

Summary

A comparison gives a boolean mask; arr[mask] keeps the True elements.
Combine conditions with &, |, ~ and brackets around each condition.
arr[mask] = value changes only the matching elements.
Fancy indexing arr[[i, j, k]] picks positions in any order; use np.ix_ for blocks.
np.where(cond, a, b) is a vectorized if/else; np.select handles many conditions.

Next: Lesson 06: Views vs Copies, reshape, ravel, flatten.