No Cardiff team had ever started the season with six successive home wins and Malky Mackay's side looked like they would fall at the final hurdle when Tommie Hoban forced the ball home from close range just before the half-hour.
Peter Whittingham's equaliser from the penalty spot (pictured) was sandwiched between red cards for Daniel Pudil and Nathaniel Chalobah as the visitors shot themselves in the foot.
But Watford battled on with nine men for 20 minutes until the first minute of added time, when Gunnarsson nodded home the winner.
Both managers made three changes after contrasting results at the weekend. Mackay was able to select Craig Bellamy after his return from injury while Craig Noone and Ben Turner were also recalled.
Matej Vydra, Watford's match-winner on Saturday, was handed a start along with Mark Yeates and Alexandre Geijo.
Despite Cardiff enjoying the best of the early exchanges, keeper David Marshall had to make a fine stop on 17 minutes to deny Vydra his second goal in four days. Teed up by Geijo, the Czech Republic international's shot was deflected and only kept out by the Scot's fingertips.
Against the run of play the away side took the lead on 27 minutes through 19-year-old Hoban. Marco Cassetti's header at the near post was kept out by Marshall and the Irishman stabbed the ball home from close range in a packed penalty area.
The home side, stung by the Watford opener, almost hit back three minutes later but Don Cowie fired into the side netting from a tight angle.
The 424 travelling fans were almost celebrating 10 minutes into the second half when a counter-attack ended with Geijo forcing Marshall into a full-stretch save. And when the resulting corner was not cleared, Fitz Hall had time to spin before clipping the top of the bar.
Watford were reduced to 10 men on 63 minutes when Pudil saw red for an off the ball incident with Noone. A minute later Heidar Helguson was presented with a great chance to draw Cardiff level when Ben Turner looped a header across goal but the former Hornet could not test Manuel Almunia.
The equaliser did arrive on 71 minutes when Matthew Connolly's shot was handled by Jonathan Hogg and up stepped Whittingham to put the ball straight down the middle for his seventh of the season.
Watford's task then became even more difficult seconds later when Chelsea loanee Chalobah was sent off. The referee blew for an offside flag against Geijo and when Chalobah kicked the ball away he picked up a second yellow card.
Nine-man Watford continued to drop deeper and had Almunia to thank for a good stop to keep out Noone. Rudy Gestede and Gunnarsson were introduced as Mackay looked to snatch a late victory against his former employers.
And just when it looked like the brave Hornets would earn a point Noone's cross was met by Gunnarsson, unmarked on the penalty spot, to make history for Mackay's men.
Source: PA
Source: PA