Since 3 is not an eigenvalue of A, the matrix A−3I2 is invertible and B is well defined. The eigenvalues 5 and 7 are distinct, so A is diagonalisable: A=PDP−1 with D=diag(5,7).
Then
A−3I2=P(D−3I2)P−1=Pdiag(2,4)P−1.
Inverting gives
(A−3I2)−1=Pdiag(21,41)P−1.
Cubing gives
B=Pdiag(81,641)P−1.
So B has eigenvalues 81 and 641. Their sum is
trB=81+641=649≈0.141.
649
Key idea. This is spectral mapping: if Av=λv, then (A−3I)−3v=(λ−3)−3v. So the eigenvalues of B are f(λ) with f(x)=(x−3)−3. The argument needs no diagonalisability.