The city is home to the Russian military headquarters that oversees the fighting in Ukraine.
Yevgeny Prigozhin’s claim in a video posted on social media is the first confirmation that he was in Rostov.
Mr Prigozhin has long feuded with the defence ministry in Moscow, and he claimed on Friday that Russian defence minister Sergei Shoigu had ordered an attack on his field camps in Ukraine.
The Wagner chief claimed that his forces had military facilities in the city under their control, including the air field.
Other videos posted on social media showed military vehicles, including tanks, on the streets outside.
Mr Prigozhin said his forces had faced no resistance from young conscripts at checkpoints and that his forces “aren’t fighting against children”.
“But we will destroy anyone who stands in our way,” he said in one of a series of angry video and audio recordings posted on social media beginning late on Friday.
“We are moving forward and will go until the end.”
Russia’s security services had responded to Mr Prigozhin’s declaration of an armed rebellion by calling for his arrest.
In a sign of how seriously the Kremlin takes the threat, security was heightened in Moscow and in Rostov-on-Don.
It is not immediately clear how Mr Prigozhin’s forces were able to enter the southern Russian city or how many troops he had with him.
Mr Prigozhin alleged that the chief of the general staff, Gen Valery Gerasimov, had scrambled warplanes to strike Wagner’s convoys, which were driving alongside ordinary vehicles.
He also said his forces had shot down a Russian military helicopter that fired on a civilian convoy, but there was no independent confirmation of this.
Mr Prigozhin said Wagner field camps in Ukraine were struck by rockets, helicopter gunships and artillery fire on orders from Gerasimov, following a meeting in Rostov with defence minister Sergei Shoigu at which they decided to destroy Wagner.
Mr Prigozhin said he had 25,000 troops under his command and would punish Mr Shoigu in an armed rebellion, and urged the army not to offer resistance. “This is not a military coup, but a march of justice,” he claimed.
While the outcome of the confrontation is still unclear, it appears likely to further hinder Moscow’s war effort as Kyiv’s forces are probing Russian defences in the initial stages of a counter-offensive.
The dispute, especially if Mr Prigozhin were to prevail, also could have repercussions for President Vladimir Putin and his ability to maintain a united front.
The Wagner forces have played a crucial role in Russia’s war in Ukraine, succeeding in taking the city where the bloodiest and longest battles have taken place, Bakhmut.
But Mr Prigozhin has increasingly criticised Russia’s military leadership, accusing it of incompetence and of starving his troops of weapons and ammunition.
On Friday, the National Anti-Terrorism Committee, which is part of the Federal Security Services (FSB), charged Mr Prigozhin with calling for an armed rebellion, punishable by up to 20 years in prison.
The FSB urged Wagner’s contract soldiers to arrest Mr Prigozhin and refuse to follow his “criminal and treacherous orders”.
It called his statements a “stab in the back to Russian troops” and said they amounted to fomenting armed conflict.
Mr Putin was informed about the situation and “all the necessary measures were being taken,” Kremlin spokesperson Dmitry Peskov said.
On Saturday morning he added that Mr Putin would address the nation “shortly”.
Heavy military trucks and armoured vehicles were seen in several parts of central Moscow early on Saturday, and soldiers were deployed outside the main building of the defence ministry. The area around the presidential administration near Red Square was also blocked, snarling traffic.