File I/O and Structured Arrays

Arrays live in memory and disappear when your program ends. In this lesson you save them to files and load them back, both in NumPy's own fast format and as CSV text that Excel and other programs understand. Then you meet structured arrays: arrays whose elements are rows with named columns of different types, which is exactly what you get when you read a CSV file with column names.

Function	File	Use for
`np.save` / `np.load`	`.npy` (binary)	one array, fast, keeps shape and dtype
`np.savez` / `np.savez_compressed`	`.npz`	several named arrays in one file
`np.savetxt` / `np.loadtxt`	text / CSV	sharing with other programs; clean data
`np.genfromtxt`	text / CSV	files with missing values or column names

How to run the examples

Create a new file named file_io_structured.py in your NumpyTutorials folder (see the setup in lesson 01). Any name works; this one just matches the topic. Start the file with these lines:

from pathlib import Path

import numpy as np

DATA = Path(__file__).parent / "data"
OUTPUT = Path(__file__).parent / "output"
OUTPUT.mkdir(exist_ok=True)

Then copy the examples of this page into the same file, one after another, in the order shown, and run it with python file_io_structured.py (with your virtual environment active). Some examples use variables created in earlier ones, so keep everything in the one file. Each Output box shows what that example prints.

The DATA line points to a data folder next to your Python file; that is where the program reads files from. The OUTPUT lines create an output folder for the files the program writes.

Create the two sample CSV files

In your project folder, create a folder named data and save these two files in it (in Notepad, choose "Save as type: All files" so the name stays .csv).

data/marks.csv: roll number and marks of 5 students:

roll_no,maths,science,english
1,78,85,92
2,88,79,95
3,65,90,81
4,72,68,74
5,95,91,89

data/sales_with_missing.csv: sales with some values missing on purpose (an empty units value, an empty price and an NA):

city,month,units,price
Pune,1,120,45.5
Mumbai,1,,52.0
Delhi,1,95,
Pune,2,130,45.5
Mumbai,2,210,52.0
Delhi,2,NA,48.0

Part A: Files

1. save and load a .npy file

scores = np.array([[78, 85, 92],
                   [88, 79, 95]])
np.save(OUTPUT / "scores.npy", scores)              # binary file, keeps shape and dtype

loaded = np.load(OUTPUT / "scores.npy")
print("loaded:\n", loaded)
print("shape and dtype kept:", loaded.shape, loaded.dtype)
print("file size:", (OUTPUT / "scores.npy").stat().st_size, "bytes")

Output

loaded:
 [[78 85 92]
 [88 79 95]]
shape and dtype kept: (2, 3) int64
file size: 176 bytes

np.save writes the raw bytes plus a small header with the shape and dtype, so np.load gives back exactly the same array. It is the fastest and safest format when only NumPy needs to read the file. The .npy file is binary, so you cannot open it in Notepad.

2. savez and savez_compressed (.npz)

days = np.arange(1, 366)
temps = np.round(25 + 8 * np.sin(days / 365 * 2 * np.pi), 1)

np.savez(OUTPUT / "weather.npz", days=days, temps=temps)             # name=array
np.savez_compressed(OUTPUT / "weather_small.npz", days=days, temps=temps)

with np.load(OUTPUT / "weather.npz") as archive:
    print("arrays inside:", archive.files)
    print("first 5 temps:", archive["temps"][:5])

print("normal size    :", (OUTPUT / "weather.npz").stat().st_size, "bytes")
print("compressed size:", (OUTPUT / "weather_small.npz").stat().st_size, "bytes")

Output

arrays inside: ['days', 'temps']
first 5 temps: [25.1 25.3 25.4 25.6 25.7]
normal size    : 6344 bytes
compressed size: 1675 bytes

np.savez(file, name=array, ...) stores several arrays in one .npz file, each under a name.
np.load on an .npz returns an archive: archive.files lists the names, and archive["temps"] reads one array. The with block closes the file afterwards.
savez_compressed zips the data. Here the file shrank from 6,344 to 1,675 bytes.

3. savetxt and loadtxt

np.savetxt(OUTPUT / "scores.csv", scores, delimiter=",", fmt="%d", header="maths,science,english", comments="")
print("scores.csv contains:")
print((OUTPUT / "scores.csv").read_text())

marks = np.loadtxt(DATA / "marks.csv", delimiter=",", skiprows=1)    # skip the header line
print("marks.csv loaded:\n", marks)

only_maths = np.loadtxt(DATA / "marks.csv", delimiter=",", skiprows=1, usecols=1, dtype=int)
print("only the maths column:", only_maths)
print("class average per subject:", marks[:, 1:].mean(axis=0).round(1))

Output

scores.csv contains:
maths,science,english
78,85,92
88,79,95

marks.csv loaded:
 [[ 1. 78. 85. 92.]
 [ 2. 88. 79. 95.]
 [ 3. 65. 90. 81.]
 [ 4. 72. 68. 74.]
 [ 5. 95. 91. 89.]]
only the maths column: [78 88 65 72 95]
class average per subject: [79.6 82.6 86.2]

savetxt writes a readable text file. delimiter="," makes it a CSV, fmt="%d" writes whole numbers (the default is long scientific notation), and header adds a first line. comments="" stops NumPy from putting # in front of the header.
loadtxt reads it back. skiprows=1 skips the header line, because text such as "roll_no" is not a number.
usecols=1 loads only one column, and dtype=int keeps the values as whole numbers.

4. genfromtxt with missing values

Real files have gaps. This file has an empty units value, an empty price and an "NA":

print((DATA / "sales_with_missing.csv").read_text())

try:
    np.loadtxt(DATA / "sales_with_missing.csv", delimiter=",", skiprows=1, usecols=(2, 3))
except ValueError as error:
    print("loadtxt error:", error)

numbers = np.genfromtxt(DATA / "sales_with_missing.csv", delimiter=",", skip_header=1,
                        usecols=(2, 3), missing_values="NA")
print("genfromtxt (missing -> nan):\n", numbers)

filled = np.genfromtxt(DATA / "sales_with_missing.csv", delimiter=",", skip_header=1,
                       usecols=(2, 3), missing_values="NA", filling_values=0)
print("with filling_values=0:\n", filled)

Output

city,month,units,price
Pune,1,120,45.5
Mumbai,1,,52.0
Delhi,1,95,
Pune,2,130,45.5
Mumbai,2,210,52.0
Delhi,2,NA,48.0

loadtxt error: could not convert string '' to float64 at row 1, column 3.
genfromtxt (missing -> nan):
 [[120.   45.5]
 [  nan  52. ]
 [ 95.    nan]
 [130.   45.5]
 [210.   52. ]
 [  nan  48. ]]
with filling_values=0:
 [[120.   45.5]
 [  0.   52. ]
 [ 95.    0. ]
 [130.   45.5]
 [210.   52. ]
 [  0.   48. ]]

loadtxt stops with an error at the first empty field.
genfromtxt is the more forgiving reader: empty fields, and anything listed in missing_values (here "NA"), become nan. Note it uses skip_header instead of skiprows.
filling_values=0 puts your own value in the gaps instead of nan.

5. genfromtxt with column names

sales = np.genfromtxt(DATA / "sales_with_missing.csv", delimiter=",", names=True,
                      dtype=None, encoding="utf-8", missing_values="NA")
print("column names:", sales.dtype.names)
print("city column :", sales["city"])
print("units column:", sales["units"], "<- missing whole numbers become -1")
print("price column:", sales["price"], "<- missing decimals become nan")
print("Pune units  :", sales["units"][sales["city"] == "Pune"])

Output

column names: ('city', 'month', 'units', 'price')
city column : ['Pune' 'Mumbai' 'Delhi' 'Pune' 'Mumbai' 'Delhi']
units column: [120  -1  95 130 210  -1] <- missing whole numbers become -1
price column: [45.5 52.   nan 45.5 52.  48. ] <- missing decimals become nan
Pune units  : [120 130]

names=True reads the column names from the first line.
dtype=None lets NumPy choose a type per column: text for city, whole numbers for units, decimals for price. encoding="utf-8" reads the text properly.
The result is a structured array: you get a column by name, sales["units"].
Watch the gaps: a missing whole number cannot be nan, so genfromtxt puts -1 there. A missing decimal becomes nan. Always check columns that may have gaps.

Part B: Structured arrays

6. Creating a structured array

A normal array has one type for everything. A structured array has a type made of several named fields, so each element is a small record, like one row of a table.

employee_type = np.dtype([
    ("name", "U10"),          # text, at most 10 characters
    ("age", "i4"),            # 32-bit integer
    ("salary", "f8"),         # 64-bit float
])

employees = np.array([
    ("Asha", 29, 95000.0),
    ("Ravi", 35, 72000.5),
    ("Neha", 41, 65000.0),
    ("Kiran", 26, 110000.0),
], dtype=employee_type)

print(employees)
print("dtype :", employees.dtype)
print("shape :", employees.shape, "(4 rows)")

Output

[('Asha', 29,  95000. ) ('Ravi', 35,  72000.5) ('Neha', 41,  65000. )
 ('Kiran', 26, 110000. )]
dtype : [('name', '<U10'), ('age', '<i4'), ('salary', '<f8')]
shape : (4,) (4 rows)

The dtype is a list of (name, type) pairs. "U10" is text of up to 10 characters, "i4" is a 4-byte integer (int32) and "f8" is an 8-byte float (float64).
Each element is written as a tuple, in the same order as the fields.
The array is 1D: shape (4,), 4 records.

7. Columns, rows, filters, sorting

print("names        :", employees["name"])
print("row 0        :", employees[0])
print("row 0 name   :", employees[0]["name"])
print("average age  :", employees["age"].mean())
print("salary > 80k :", employees[employees["salary"] > 80000]["name"])

employees["salary"] = np.round(employees["salary"] * 1.10, 2)     # 10% raise for everyone
print("after raise  :", employees["salary"].round(2))

print("sorted by age:", np.sort(employees, order="age")["name"])
print("youngest     :", employees[employees["age"].argmin()])

Output

names        : ['Asha' 'Ravi' 'Neha' 'Kiran']
row 0        : ('Asha', 29, 95000.0)
row 0 name   : Asha
average age  : 32.75
salary > 80k : ['Asha' 'Kiran']
after raise  : [104500.    79200.55  71500.   121000.  ]
sorted by age: ['Kiran' 'Asha' 'Ravi' 'Neha']
youngest     : ('Kiran', 26, 121000.0)

employees["name"] gives a whole column as a normal array, so all the usual maths works on it.
employees[0] gives a record (one row), and employees[0]["name"] gives one value.
Masks work as usual: employees[employees["salary"] > 80000] keeps whole records.
Assigning to a column changes it for every record.
np.sort(..., order="age") sorts the records by a field.

8. Save and load a structured array

np.save(OUTPUT / "employees.npy", employees)
back = np.load(OUTPUT / "employees.npy")
print("same column names after loading:", back.dtype.names)
print(back)

Output

same column names after loading: ('name', 'age', 'salary')
[('Asha', 29, 104500.  ) ('Ravi', 35,  79200.55) ('Neha', 41,  71500.  )
 ('Kiran', 26, 121000.  )]

.npy files keep the field names and types too, so a structured array comes back exactly as it was saved.

Structured arrays are great for small, fixed-format tables and binary data. For real data analysis with many columns, filtering, grouping and joining, the next step is pandas, which is built on NumPy and makes these table operations much easier.

Exercises

Try these exercises on your own before moving on:

Save np.arange(100).reshape(10, 10) to a .npy file and load it back.
Save the marks table as marks_copy.csv with savetxt (fmt="%.1f").
Load only the english column of data/marks.csv.
Make a structured array of products (code U5, price f8, stock i4) and print the products with stock below 10.

Summary

np.save/np.load store one array in a fast binary .npy file, keeping shape and dtype.
np.savez/savez_compressed store several named arrays in one .npz file.
np.savetxt/np.loadtxt read and write CSV; use skiprows, usecols, fmt.
np.genfromtxt handles missing values (missing_values, filling_values) and column names.
Structured arrays hold records with named fields of different types: arr["field"] gives a column.

Next: Lesson 16: Performance, Memory Layout and Strides.