It was definitely a foul on the keeper but you are right, he should have got his fingers on it and got a touch on it at least.
Brings back a bitter memory for me of something similar as I went up to catch a high ball and as I had the ball in my hands I got a strong nudge on my back and lost the ball but still thinking too much about trying to catch the ball I forgot about my landing and crash landed on my ankle and twisted it badly, the loose ball was tapped in for a goal and I walked off the field limping and left leaking my wounds for the next couple of months, no foul either!