Police say they are investigating whether the 66-year-old suffered a medical episode behind the wheel.
The incident occurred when an SUV mounted a curb and hit patrons outside a pub in Daylesford, about 110km (68 miles) northwest of Melbourne, on Sunday evening, police in the state of Victoria said on Monday.
Four people died at the scene, while a girl died in hospital after being airlifted to Melbourne for treatment, police said.
Five other people, including an 11-month-old boy, are in hospital being treated for injuries.
Victoria Police Chief Commissioner Shane Patton said the car had collided with a “large number of people” at tables outside the pub.
“It’s one of the most confronting scenes, I’ve been told, that experienced officers have been to,” Patton said during a news conference.
Patton said police had yet to speak to the 66-year-old driver of the vehicle as he was being treated in hospital and the cause of the incident was still unclear.
He said the driver had been breath-tested and had no alcohol in his system, and that police were investigating all avenues, including the possibility the driver suffered a medical incident behind the wheel.