Vectors and Matrices

##Vectors and Matrices – The Building Blocks

What Are Vectors?

A vector is an ordered list of numbers representing a point or direction in space. In Machine Learning, vectors represent data points, features, or model parameters.

Mathematical Notation: A vector with n elements is written as:

v = [v₁, v₂, v₃, ..., vₙ]

ML Context: Each row in your dataset is a feature vector—a collection of measurements describing one sample.

import numpy as np

# A feature vector representing a house
# [square_feet, bedrooms, bathrooms, age_years]
house_features = np.array([1500, 3, 2, 10])

print(f"Feature vector: {house_features}")
print(f"Number of features (dimensions): {house_features.shape[0]}")

This vector represents one house with four features. The entire dataset would contain many such vectors.

Vector Operations

Addition and Subtraction: Vectors of the same size can be added element-wise.

import numpy as np

# Two feature vectors
vector_a = np.array([1, 2, 3])
vector_b = np.array([4, 5, 6])

# Vector addition
sum_vector = vector_a + vector_b
print(f"Addition: {sum_vector}")  # [5, 7, 9]

# Vector subtraction
diff_vector = vector_a - vector_b
print(f"Subtraction: {diff_vector}")  # [-3, -3, -3]

Scalar Multiplication: Multiplying a vector by a single number (scalar) scales all elements.

import numpy as np

vector = np.array([2, 4, 6])
scalar = 3

scaled_vector = scalar * vector
print(f"Scaled vector: {scaled_vector}")  # [6, 12, 18]

In ML, scalar multiplication is used in feature scaling and gradient updates.

Dot Product

The dot product is one of the most important operations in Machine Learning. It multiplies corresponding elements and sums the results.

Formula:

a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ

import numpy as np

# Feature vector and weight vector
features = np.array([1500, 3, 2])  # house features
weights = np.array([100, 10000, 5000])  # learned weights

# Dot product computes weighted sum (prediction)
prediction = np.dot(features, weights)
print(f"Predicted price: ${prediction:,}")
# 1500*100 + 3*10000 + 2*5000 = $190,000

The dot product is the core operation in linear regression, neural networks, and many other ML algorithms. It computes how much each feature contributes to the prediction.

Vector Magnitude and Normalization

Magnitude (Length): The magnitude measures the "size" of a vector.

import numpy as np

vector = np.array([3, 4])

# Calculate magnitude (Euclidean norm)
magnitude = np.linalg.norm(vector)
print(f"Magnitude: {magnitude}")  # 5.0 (3² + 4² = 25, √25 = 5)

Normalization: Normalizing converts a vector to unit length (magnitude = 1), preserving direction.

import numpy as np

vector = np.array([3, 4])

# Normalize to unit vector
unit_vector = vector / np.linalg.norm(vector)
print(f"Unit vector: {unit_vector}")  # [0.6, 0.8]
print(f"New magnitude: {np.linalg.norm(unit_vector)}")  # 1.0

Normalization is essential for algorithms sensitive to feature scales, like k-nearest neighbors and neural networks.

What Are Matrices?

A matrix is a two-dimensional array of numbers arranged in rows and columns. In Machine Learning, matrices represent entire datasets, weight matrices in neural networks, and transformation operations.

Mathematical Notation: A matrix with m rows and n columns:

A = | a₁₁  a₁₂  a₁₃ |
    | a₂₁  a₂₂  a₂₃ |

import numpy as np

# Dataset as a matrix: 4 samples, 3 features each
dataset = np.array([
    [1500, 3, 2],   # House 1
    [1200, 2, 1],   # House 2
    [1800, 4, 2],   # House 3
    [2000, 4, 3]    # House 4
])

print(f"Dataset shape: {dataset.shape}")  # (4, 3) = 4 rows, 3 columns
print(f"Number of samples: {dataset.shape[0]}")
print(f"Number of features: {dataset.shape[1]}")

Special Matrices

Identity Matrix: A square matrix with 1s on the diagonal and 0s elsewhere. Multiplying by the identity matrix leaves data unchanged.

import numpy as np

# 3x3 identity matrix
I = np.eye(3)
print("Identity matrix:")
print(I)
# [[1. 0. 0.]
#  [0. 1. 0.]
#  [0. 0. 1.]]

# Any matrix times identity equals itself
A = np.array([[1, 2], [3, 4]])
print(f"A @ I = A: {np.allclose(A @ np.eye(2), A)}")  # True

Transpose: Flipping a matrix over its diagonal—rows become columns.

import numpy as np

matrix = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

transposed = matrix.T
print("Original (2x3):")
print(matrix)
print("\nTransposed (3x2):")
print(transposed)

Transposition is used frequently in ML computations, especially when aligning matrix dimensions for multiplication.

Vectors and Matrices in ML Context

Concept	ML Application
Feature Vector	One data sample's measurements
Weight Vector	Model parameters to learn
Data Matrix	Entire dataset (rows = samples, columns = features)
Dot Product	Computing predictions, similarity
Normalization	Feature scaling, regularization