Types of matrices

Linear Algebra

Some matrix operations and theory apply to special types of matrices, so it is good to be aware of them.

In this article, we will try to provide a comprehensive overview of the various types of matrices. We will also comment on the desirability of certain matrix types.

Prerequisites

To understand types of matrices, we recommend familiarity with the concepts in

Follow the above links to first get acquainted with the corresponding concepts.

Square and rectangular matrices

A square matrix is one with an equal number of rows and columns. Matrices with unequal rows and columns are known as rectangular, by the virtue of their shape.

Symmetric matrix

A symmetric matrix is one with the exact same entries on either side of the main diagonal.

That means \( \mA \) is symmetric if and only if \( \va_{i,j} = \va_{j,i} \) for all \( i, j \in \{1,\ldots,n \} \).

Needless to say, a matrix is symmetric only if it is also square.

Diagonal matrix

A diagonal matrix is a square symmetric matrix one with non-zero elements along the main diagonal and zeros elsewhere.

It is usually denoted as \( \text{diag}(d_1,\ldots,d_n) \).

Identity matrix

An identity matrix is a diagonal matrix that has \( 1 \)'s along the main diagonal and \( 0 \)s everywhere else.

It is usually denoted as \( \mI \), such that

\begin{equation} \mI_{i,j} = \begin{cases} 1 & \text{ , if } i = j \\ 0 & \text{ , if } i \neq j \end{cases} \end{equation}

Triangular matrices

An upper triangular matrix is one which has non-zero elements along the main diagonal and above the main diagonal. All other elements are zero.

Conversely, a lower triangular matrix is one which has zeros above the main diagonal and non-zero elements elsewhere.

Orthogonal matrix

A matrix with mutually orthonormal rows and columns is known as an orthogonal matrix.

Matrices by definiteness

There are also classes of matrices based on their definiteness, a general concept in mathematics for quadratic forms. So, a square symmetric matrix \( \mA \in \real^{n \times n} \) is called

  • positive definite if \( \vx^T \mA \vx > 0, \forall \vx \in \real^n \).
  • positive semi-definite if \( \vx^T \mA \vx \ge 0, \forall \vx \in \real^n \).
  • negative definite if \( \vx^T \mA \vx < 0, \forall \vx \in \real^n \).
  • negative semi-definite if \( \vx^T \mA \vx \le 0, \forall \vx \in \real^n \).
  • indefinite if \( (\vx^T \mA \vx)(\vy^T \mA \vy) < 0 \), for some \( \vx \in \real^n \) and \( \vy \in \real^n \).

Hermitian and unitary matrices

A Hermitian matrix is the complex analogue of real symmetric matrix.

It is a complex square matrix that is equal to its own conjugate transpose.

A Unitary matrix is the complex analogue of real orthogonal matrix.

The good, the bad, and the ugly matrix

You will notice in most machine learning literature, that the authors squeal with delight if they are working with certain matrices. Why? Because, as you have noticed so far, certain operations, especially inversion, becomes possible, sometimes easier, for some types of matrices.

First, let's talk about the ugliest of all. A rectangular matrix that is also singular. We cannot invert it. Eigendecomposition is not defined for it. Only thing we can definitely do is a singular value decomposition on it.

Among bad matrices, there are levels of bad. The worst are those that are singular, even if they are square or symmetric. The better ones are non-singular matrices that are also symmetric or at least square. They are invertible and also have an eigendecomposition.

The best matrices are the ones that are easier to work with. Identity, diagonal, orthogonal, and positive definite matrices are the best because of their rich properties and theory.

So, whenever you encounter a matrix, be on the lookout for some of these properties. If you are not dealing with a desirable one, try to take the extra effort to bring it closer to a better one.

Check out this next chart which shows a rough matrix hierarchy to identify which ones are invertible, symmetric, and positive definite. It is not a Venn-diagram in the true sense, but just shows containment. For example, all orthogonal matrices are invertible.

Relationships among matrix types

The following figure summarizes important relationships among the popular matrix types.

Relationships among matrix types

Where to next?

In this section, we have merely defined the various matrix types. To really build intuition about what these actually mean, we first need to understand the effect of multiplying a particular type of matrix. We present this in matrix as a transformer.

You may also choose to explore other advanced topics linear algebra.

Already feeling like an expert in linear algebra? Move on to other advanced topics in mathematics or machine learning.

Please support us

Help us create more engaging and effective content and keep it free of paywalls and advertisements!

Let's connect

Please share your comments, questions, encouragement, and feedback.