Australian police used pepper spray to break up clashes between rightwing nationalists and anti-racism protesters on the streets of Melbourne on Saturday as Prime Minister Malcolm Turnbull said eliminating racism was still a "work in progress".

The United Patriots Front (UPF) organised a "Stop the Far Left" rally while anti-racism protesters marched under the banner "Moreland says No to Racism", in reference to a Melbourne district.

Fears of young Australian Muslims being inspired by militants such as Islamic State and travelling to fight in Iraq and Syria have underpinned support for rightwing groups like UPF and Reclaim Australia.

Former prime minister Tony Abbott made blocking migrants trying to reach Australia by boat a key plank of his victorious election campaign in 2013.

TV footage showed UPF members with Australian flags on poles and their faces covered by bandanas wrestling with leftwing militants, many of whom were hooded and had black cloths over their faces.

On one occasion, a UPF protester who fell to the ground was kicked several times by two anti-racism activists.

"I understand the need and the right to protest ... but what we can't have is this violence in our community," Victorian state Police Commander Sharon Cowden told Sky. "We saw inappropriate and often cowardly behaviour, people wearing masks."

ABC said two people were arrested for weapons offences before the trouble flared despite a large police presence.

Two people were arrested for rioting, one for assaulting police, one for hindering police and another for robbery, ABC said.

Turnbull, responding to comments by opposition leader Bill Shorten that racism was rife in Australia, said Australia was not "entirely" blameless.

"But I have to say we are the most successful multicultural society in the world," Turnbull told reporters. "There is more work to do. This is not something to be complacent about. This is a work in progress."