Yes, he is pushing you away, as he is trying to minimise the pain you are suffering but in doing so does not realise that you are suffering.
Does stage 4 prostrate cancer mean he has a high change of dying? If so, what was the time limit?
Perhaps he just wants to enjoy the last of his life his way.
Remind him you are there for better or worse and that you would prefer to create wonderful memories of the time you are together. Take trips (outings, day trips) places that hubby would like to see, like to do.
Perhaps a visit to a doctor/social worker (who is older) and discuss the issues.
It is not easy when a loved one lashes out at us, but they do it, because they trust us, they know that at the end of the day, we still love them.
Get the meds checked, as some can effect behaviour.
Good luck