He insisted the Lilywhites had matched the unbeaten league leaders right up until they were awarded a penalty when Dean Grant was brought down.
The match hinged on that crucial moment - with the referee changing his mind after initially showing a red card as well as pointing to the spot.
Instead, he gave the Tooting player a yellow - and minutes after Town missed the penalty, the visitors grabbed the lead at the other end and the game swung in their favour.
But Ray backed the referee for changing his mind and insisted the Lilywhites lacklustre response to falling behind was out of character.
Speaking in an extended View From The Dugout video interview after the game, he said: "It was a pretty clear penalty. The referee got it right.
"You can't have the double jeopardy this year, so he changed the red to a yellow and we had the penalty.
"I've got no problem with how he went about it. It was a great opportunity to go in front in a tight game and the whole game seems to have hinged on that fairly critical moment.
"We're all very disappointed. I'm particularly disappointed with the way we've reacted to going behind.
"It's unlike us. We needed to just show more resolve and get ourselves back into the game.
"But up to that moment you wouldn't have separated the sides, and the scoreline is a big disappointment.
"I think we were much the stronger side in the first half. I think we lacked a little bit of intensity if I'm honest.
"I think we lacked a little bit of energy, which may have been a hangover from Monday.
"No excuses. There never is from me. I don't think we were good enough in the last 20 minutes and ultimately we paid the price for not taking our chance first."
Hear what else Ray had to say about the Tooting match and Tuesday's upcoming game against Chatham by clicking on the link below to watch the whole extended video.