Guilsfield could not hold on to a 2-0 half time lead as Caernarfon snatched victory in the final minute.
The game was played at a frantic pace from start to finish with the visitors starting the match on the front foot, pressurising the home defence and keeping Dave Littleford in goal very busy.
However, Guilsfield took the lead on 25" with their first meaningful attack after a quick break from Mark Davies down the left who found Chris Cathrall in the area and his neat touch to Gareth Jones led to the village boy crashing home. The goal gave Guilsfield the impetus to go on the attack and a series of flowing moves followed. Then on 32" Rhydian Morris beat keeper to the ball and his header found the net with Guilsfield well on top at the break.
The second half began in much the same fashion, with Morris' snap shot turned away by Pritchard in the first minute. On 56" Morris had a chance to bury the match but just failed to get his shot away. Just two minutes later, Caernarfon gained a foothold in the game after a dubious free kick was awarded and the set piece led to Morris' header sneaking in at the far post for an unlucky own goal.
Caernarfon tails were up now and they gained the ascendancy with the majority of the possession. Their equaliser came from new signing McArdle who scored with a left footed strike on 79".
There was another twist on 81" as Guilsfield were awarded a penalty for handball. With usual penalty taker Danny Barton not playing, Chris Cathrall stepped up but unfortunately his shot struck the crossbar and back into play.
The game looked to be heading for extra time, with Caernarfon having already used all their outfield subs, but in a final moment in this cup tie a scramble in the Guilsfield area led to the ball falling to Austin who found the net.
Guilsfield, in a team without top scorer Danny Barton, captain Nathan Leonard and James O'Reilly, gave the Canaries a tough game but the spoils go to the visitors who enter the hat for the second round.