CS 598 EVS: Tensor Computations, Lecture notes of Linear Algebra

Download CS 598 EVS: Tensor Computations and more Lecture notes Linear Algebra in PDF only on Docsity!

CS 598 EVS: Tensor Computations

Course Overview

Edgar Solomonik

University of Illinois at Urbana-Champaign

Tensors

A tensor is a collection of elements I (^) its dimensions define the size of the collection I (^) its order is the number of different dimensions I (^) specifying an index along each tensor mode defines an element of the tensor A few examples of tensors are I (^) Order 0 tensors are scalars, e.g., s ∈ R I (^) Order 1 tensors are vectors, e.g., v ∈ Rn I (^) Order 2 tensors are matrices, e.g., A ∈ Rm×n I (^) An order 3 tensor with dimensions s 1 × s 2 × s 3 is denoted as T ∈ Rs^1 ×s^2 ×s^3 with elements tijk for i ∈ { 1 ,... , s 1 }, j ∈ { 1 ,... , s 2 }, k ∈ { 1 ,... , s 3 }

Tensor Contractions

A tensor contraction multiplies two tensors to produce a third I (^) Examples: inner product, outer product, tensor product, Hadamard (elementwise) product, matrix multiplication I (^) One higher order example is tensor-times-matrix (TTM), e.g.,

tijkl =

q

uijqlvqk

I (^) A common contraction between two high order tensors is

tabij =

p,q

uapiqvpbqj

I (^) Tensor contractions can be reduced to products of matrices and/or vectors by transposing modes and matricizing both operands, then folding and transposing the product

Tensor Contraction Expressions

In most applications, we wish to evaluate mathematical expressions involving contraction of more than two tensors I (^) Contractions of more than two tensors are also sometimes referred to as einsums (short for Einstein summation, in reference to Einstein’s convention for omitting summation indices) I (^) One important example is the ‘MTTKRP’ kernel

uir =

j,k

tijkvjrwkr

I (^) Efficient evaluation of such kernels requires specialized algorithms I (^) Contraction algorithms must also be adapted to leverage tensor properties such as symmetry with respect to permutation of modes, block-wise group symmetries, and data sparsity

Applications of Tensor Decompositions

I (^) Tensor decompositions provide a mechanism for approximating tensor datasets with a smaller number of degrees of freedom I (^) polynomial improvements are obtained in electronic structure calculations I (^) exponential improvements are obtained for representing some quantum states

I (^) With imposition of constraints (e.g., nonnegativity or orthogonality), they can be used for data mining tasks such as high-order clustering I (^) in the presence of missing data, tensor decompositions may be used to perform tensor completion

I (^) When the tensor represents an operator or mapping, tensor decompositions can be used to find reduced structure I (^) fast algorithms, such as FFT and Strassen’s matrix multiplication algorithm, may be viewed as tensor decompositions

Tensor Decomposition Theory

I (^) Many basic decomposition/approximation problems are formally NP-hard I (^) A considerable amount of theory focuses on CP decomposition and CP rank, some will be surveyed in this course I (^) A few alternate notions of tensor eigenvalues and singular values exist, and may be loosely tied to decompositions I (^) Stability and conditioning results exist for the tensor as an operator and CP decomposition as a problem

Tensor Networks

I (^) Tensor network methods take as input a tensor that is already decomposed I (^) Goal is generally to learn something about an operator described by a tensor network I (^) Often want to compute extremal eigenpairs of matrix M a tensor folding of which T is described by the tensor network, e.g.,

M = A ⊗ B + C ⊗ D

I (^) Unknowns, e.g., eigenvectors in eigenproblem above, often also represented implicitly by a tensor decomposition I (^) These methods are prevalent for studying quantum systems, which involve a Hamiltonian acting on a space that is exponential in size relative to the system I (^) In this context, tensor networks are also effective for time-evolution

Tensor Network Theory and Algorithms

I (^) Different classes of functions have low rank with respect to different tensor networks I (^) 1D and 2D tensor networks are most widely used for quantum systems I (^) Successive (alternating) quadratic optimization also widely used for tensor networks I (^) Canonical forms propagate orthogonality conditions to ensure stability I (^) Naive contraction of 2D tensor networks has exponential cost, various approximate algorithms exist I (^) Other tensor networks trade-off connectivity and contractibility