Higham, Nicholas J. and Pranesh, Srikara and Zounon, Mawussi (2018) Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems. [MIMS Preprint]
There is a more recent version of this item available. 
Text
paper.pdf Download (500kB) 
Abstract
Motivated by the demand in machine learning, modern computer hardware is increas ingly supporting reduced precision floatingpoint arithmetic, which provides advantages in speed, energy, and memory usage over single and double precision. Given the availability of such hardware, mixed precision algorithms that work in single or double precision but carry out part of a compu tation in half precision are now of great interest for general scientific computing tasks. Because of the limited range of half precision arithmetic, in which positive numbers lie between 6 × 10−8 and 7 × 104, a straightforward rounding of single or double precision data into half precision can lead to overflow, underflow, or subnormal numbers being generated, all of which are undesirable. We develop an algorithm for converting a matrix from single or double precision to half precision. It first applies twosided diagonal scaling in order to equilibrate the matrix (that is, to ensure that every row and column has ∞norm 1), then multiplies by a scalar to bring the largest element within a factor θ ≤ 1 of the overflow level, and finally rounds to half precision. The second step ensures that full use is made of the limited range of half precision arithmetic, and θ must be chosen to allow sufficient headroom for subsequent computations. We apply the new algorithm to GMRESbased iterative re finement (GMRESIR), which solves a linear system Ax = b with single or double precision data by LU factorizing A in half precision and carrying out iterative refinement with the correction equations solved by GMRES preconditioned with the low precision LU factors. Previous implementations of this algorithm have used a crude conversion to half precision that our experiments show can cause slow convergence of GMRESIR for badly scaled matrices or failure to converge at all. The new conversion algorithm computes ∞norms of rows and columns of the matrix and its cost is negligible in the context of LU factorization. We show that it leads to faster convergence of GMRESIR for badly scaled matrices and thereby allows a much wider class of problems to be solved.
Item Type:  MIMS Preprint 

Subjects:  MSC 2010, the AMS's Mathematics Subject Classification > 65 Numerical analysis 
Depositing User:  Dr Srikara Pranesh 
Date Deposited:  19 Mar 2019 11:28 
Last Modified:  19 Mar 2019 11:28 
URI:  http://eprints.maths.manchester.ac.uk/id/eprint/2690 
Available Versions of this Item

Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems. (deposited 28 Nov 2018 11:51)
 Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems. (deposited 19 Mar 2019 11:28) [Currently Displayed]
Actions (login required)
View Item 