Cray made a lively start and the visiting 'keeper Craig Stoner just managed to push away a through ball from Matt Pooley as Leigh Bremner closed in.
Cray's first corner-kick was cleared with some difficulty by the Bognor defence, whereupon Alex Stavrinou cracked in a rising shot that slammed against the crossbar.
Bognor's forwards had already shown a capability to stretch the Cray defence, and on 13 minutes a pin-point cross from the right was headed home by Ashley Robinson.
Cray were nearly on terms immediately afterwards when Michael Power's free-kick was deflected into the path of Tyrone Sterling, but his close-range shot was blocked by Stoner - a fine reflex save inside the six-yard box.
An equaliser from Cray was soon to follow, thanks to the referee deciding there was a deliberate handball before Power's cross was cleared by Bognor in the 18th minute.
Bremner netted from the penalty-kick and then went close on two occasions soon afterwards.
An overhead first-time volley from Bremner sent the ball skimming over the bar when he met a Cray free-kick from the right, and then after breaking through on the left he stabbed the ball past Stoner but saw Ben Andrews clear his shot off the line.
The dangerous Harvey White skipped through the Cray defence and was unlucky to see his shot come back off the post, but Cray were not to survive another attack by Bognor that saw James Crane eventually put the ball in the net in the 42nd minute after two earlier shots had both struck the woodwork during an almighty scramble in front of the Cray goal.
Hard though Cray fought in the second half, after Bremner had headed wide from an early free-kick the visitors looked solid in defence and continued to play some good pass-and-move football on the break.
On 72 minutes Whyte got the goal that he had been threatening to score all night. He marked his return to the Rocks from Havant by netting past Andy Walker after another solo run.
Cray: Walker, Burgess, Pooley, Saunders, Willy, Sterling, Fray, Stavrinou, Bremner, Power (Parkinson 73), Clark (Phillips 73).
Subs not used: Taylor, Fasunan, Estrada