Some fans have raised concerns that the late kick off could lead to problems caused by excessive drinking.
The most recent derby matches have kicked off around midday, partly to give fans less drinking time.
The sell-out match at Cardiff City Stadium is crucial for both clubs in their push for promotion - which could lead to them playing in the Premier League next season.
In the past, matches between the two clubs have been marred by trouble, although South Wales Police say plans are in place to ensure the day is policed "efficiently and effectively".
Divisional Commander for Cardiff, Ch Supt Josh Jones, said: "My message to anyone thinking about causing trouble is don't forget about the potential that your team may be promoted to the Premier League or play each other in the play-offs.
"If you come to this event and misbehave, we will have no hesitation in banning you, which means you may miss a very important fixture in May, or miss the likes of Liverpool or Manchester United visiting south Wales next season."
Pubs around the ground are open but will only contain Cardiff fans as the 1,900 Swansea supporters are escorted straight in and straight out on official coaches.
Paul Morris, a former member of the Swansea City Supporters' Trust board, said he was concerned fans would be drinking all day.
"For Cardiff fans, they will be in the pubs all day. You've got Man United and Chelsea playing on the box earlier - that's a great excuse to start the day [drinking]," he said.
"People will be tanked up. Security is such that supporters can't really get at each other but it doesn't make for a good thing at all."
Mr Morris, a season ticket holder, said he would be watching the match on television.
Vince Alm, of Cardiff City Supporters' Club, said he was "surprised" at the kick-off time but believed the new stadium would be easier to police.
"I'm not sure 5.20pm is the best time because people will have had a good time in the pubs," he said.
"Some people do behave differently after a few drinks - that's a problem in society generally."
He added police were experienced at dealing with big derby games but he was concerned about the possibility of an incident such as somebody running on the pitch.
South Wales Police said in a statement: "Historically, South Wales Police has always sought an early kick off. However, on this occasion, Sky and the club have requested a kick-off time of 5.20pm which we have agreed to facilitate.
"We have a good working relationship with Cardiff City and continue to make strides towards normality in relation to issues such as kick-off times and transporting supporters to games."
Cardiff City said in a statement: "The design of the new stadium, control measures for visiting supporters, and confidence in the ability of stewards and police has enabled the club to facilitate a 5.20pm kick off and the game will be shown live on Sky TV.
"Cardiff City Football Club works closely with South Wales Police to enforce the ground regulations and it should be noted that at this fixture particular attention will be made to any alcohol-related offences."
Source: BBC Sport
Source: BBC Sport