I am wondering why you don't use IIR filters in the HRTF processes? This would save enormous amounts of computation and reduce bus loading and undoubtedly allow better matching to the actual HRTF-s...

We do use IIR filters in our own implementation. However, every other major spatializer (such as Oculus or Cardboard) does convolution in the frequency domain.