People threw punches, hurled water bottles and used pepper spray during the violence as riot police were deployed to try to disperse the crowds. At least eight people were hurt.
Among the three dead were two police officers who were on a helicopter that crashed near the scene.
The other death was a 32-year-old woman who had been crossing the street when she was struck by a car that hit a group of anti-Nazi campaigners also leaving 19 people injured.
The male driver has been arrested.
The rally by far-right activists was organised to protest against Charlottesville’s decision to remove a statue of Confederate General Robert E. Lee from the city.
Speaking about the disorder, US President Donald Trump said he condemned “in the strongest possible terms” what he called an “egregious display of hatred, bigotry and violence on many sides”.
He added: “What is vital now is a swift restoration of law and order and the protection of innocent lives.”
Charlottesville mayor Michael Signer had said he was disgusted the white nationalists had come to his town and blamed the US leader for inflaming racial prejudices during his presidential campaign last year.
Mr Signer said: “I’m not going to make any bones about it. I place the blame for a lot of what you’re seeing in America today right at the doorstep of the White House and the people around the president.”
The fighting prompted Charlottesville city manager Maurice Jones and interim county executive Doug Walker to simultaneously declare local states of emergency for the two jurisdictions.
The city hall then declared a rally that had led to large numbers of far-right activists gathering an “unlawful assemblage” and Virginia governor Terry McAuliffe declared an emergency as police struggled to manage the two groups.
Up to 6,000 white nationalists had been expected to attend the “Unite the Right” rally.
TV pictures showed one of the fights erupting when the anti-Nazi protesters approached the white supremacists with what looked like a wooden banner.
As they did so, they were met by a wall of men dressed in body armour and helmets, some of whom used pepper spray, police said.
Other footage showed rivals on either side using hand-made shields as they fought each other.
Initially it appeared as though police were not getting involved but the the Virginia State Police have since posted pictures of heavily armed officers carrying out arrests.
The violence prompted First Lady Melania Trump to call for an end to the clashes, tweeting: “Our country encourages freedom of speech, but let’s communicate w/o hate in our hearts. No good comes from violence. #Charlottesville”
The clashes came after a federal judge ordered Charlottesville authorities to allow a weekend rally of white nationalists and other extremists to take place.
Right-wing blogger Jason Kessler filed a lawsuit demanding the right to hold a protest at the spot but after the city said the rally must be moved out of Emancipation Park to a larger one, a US District Judge granted permission.
The Southern Poverty Law Center said it was expecting a range of far-right groups to attend including the National Socialist Movement, the League of the South, as well as “various anti-government, Patriot, militia, Klan, III Percenters and anti-immigration groups”.
The state of emergency allows local officials to request additional resources if needed to respond to ongoing events, authorities said.
Some commentators have described how many far-right activists in the US regard Charlottesville as a test case, because of what they see as an attack on parts of their history.
Two organisations that track extremist groups said the rally has the potential to be the largest of its kind in at least a decade,