Far-right activists organised the event on Friday to protest against Charlottesville’s plans to remove a statue of Confederate General Robert E Lee from the city.
But it turned ugly, as mass brawls broke out between them and anti-fascists, with people throwing punches, hurling water bottles and using pepper spray as riot police were deployed.
Virginia Governor Terry McAuliffe said: “I have a message to all the white supremacists and the Nazis who came into Charlottesville today.
“Our message is plain and simple: Go home.
“You are not wanted in this great commonwealth.
“Shame on you.
“You pretend that you are patriots but you are anything but a patriot.”
His words came after it was confirmed that three people had died in events linked to the violence and dozens had been injured.
Heather Heyer, 32, who had been crossing the street, died when she was struck by a car driving at high speed that also hit a group of anti-Nazi campaigners, injuring dozens.
James Alex Fields Jr, 20, of Ohio, has been charged with second-degree murder, three counts of malicious wounding, and one count related to leaving the scene.
Lieutenant H. Jay Cullen and Trooper-pilot Berke M.M. Bates also died after a Virginia State Police helicopter that had been assisting with policing the rally crashed.
Mr McAuliffe continued, speaking to the white supremacists: “You came here today to hurt people and you did hurt people.
“But my message is clear: we are stronger than you. You will not succeed. There is no place for you here. There is no place for you in America.”
His words came after US President Donald Trump condemned what he described as an “egregious display of hatred, bigotry and violence on many sides”.
He added: “What is vital now is a swift restoration of law and order and the protection of innocent lives.”
Some criticised Mr Trump for his seeming reluctance to specifically blame the white supremacists.
Republican senator Cory Gardner said: “We must call evil by its name. These were white supremacists and this was domestic terrorism.”
Democratic senator Adam Schiff said: “(The President) needs to speak out against the poisonous resurgence of white supremacy – there are not ‘many sides’ here, just right and wrong.”
Charlottesville mayor Michael Signer also had criticism for Mr Trump, blaming the President for inflaming racial prejudices during his presidential campaign last year.
Mr Signer said: “I’m not going to make any bones about it. I place the blame for a lot of what you’re seeing in America today right at the doorstep of the White House and the people around the President.”
The FBI announced it has opened a civil rights investigation into the deadly car ramming, saying it “will collect all available facts and evidence”.
US attorney general Jeff Sessions said: “When such (violent) actions arise from racial bigotry and hatred, they betray our core values and cannot be tolerated.”
About 6,000 white supremacists were expected to have attended the Charlottesville rally.