Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems

Higham, Nicholas J. and Pranesh, Srikara and Zounon, Mawussi (2018) Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems. [MIMS Preprint]

Text paper.pdf Download (500kB)

Abstract

Motivated by the demand in machine learning, modern computer hardware is increas- ingly supporting reduced precision floating-point arithmetic, which provides advantages in speed, energy, and memory usage over single and double precision. Given the availability of such hardware, mixed precision algorithms that work in single or double precision but carry out part of a compu- tation in half precision are now of great interest for general scientific computing tasks. Because of the limited range of half precision arithmetic, in which positive numbers lie between 6 × 10−8 and 7 × 104, a straightforward rounding of single or double precision data into half precision can lead to overflow, underflow, or subnormal numbers being generated, all of which are undesirable. We develop an algorithm for converting a matrix from single or double precision to half precision. It first applies two-sided diagonal scaling in order to equilibrate the matrix (that is, to ensure that every row and column has ∞-norm 1), then multiplies by a scalar to bring the largest element within a factor θ ≤ 1 of the overflow level, and finally rounds to half precision. The second step ensures that full use is made of the limited range of half precision arithmetic, and θ must be chosen to allow sufficient headroom for subsequent computations. We apply the new algorithm to GMRES-based iterative re- finement (GMRES-IR), which solves a linear system Ax = b with single or double precision data by LU factorizing A in half precision and carrying out iterative refinement with the correction equations solved by GMRES preconditioned with the low precision LU factors. Previous implementations of this algorithm have used a crude conversion to half precision that our experiments show can cause slow convergence of GMRES-IR for badly scaled matrices or failure to converge at all. The new conversion algorithm computes ∞-norms of rows and columns of the matrix and its cost is negligible in the context of LU factorization. We show that it leads to faster convergence of GMRES-IR for badly scaled matrices and thereby allows a much wider class of problems to be solved.

Item Type:	MIMS Preprint
Subjects:	MSC 2010, the AMS's Mathematics Subject Classification > 65 Numerical analysis
Depositing User:	Dr Srikara Pranesh
Date Deposited:	19 Mar 2019 11:28
Last Modified:	19 Mar 2019 11:28
URI:	https://eprints.maths.manchester.ac.uk/id/eprint/2690

Available Versions of this Item

• Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems. (deposited 28 Nov 2018 11:51)
  • Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems. (deposited 19 Mar 2019 11:28) [Currently Displayed]
    • Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems. (deposited 06 Jun 2019 12:46)

Actions (login required)

View Item