At each prism face, measure incidence and refraction angles from the normal, not the surface. Two refractions change the emergent ray’s direction, giving deviation from the original direction.
What causes dispersion? Refractive index varying with wavelength.
Why does a star twinkle? Changing atmospheric refraction alters the received light from its nearly point-like image.