The Country Fire Authority (CFA) confirmed 35 homes had been lost in Wye River, and 18 at Separation Creek on Friday.
Hundreds of firefighters and dozens of tankers and aircraft have been battling the blaze, which started last weekend but flared up in hot, blustery conditions.
The fire, which has so far burned more than 2,000 hectares, is not yet under control.
An emergency warning is still in place for Wye River, Separation Creek, Kennett River and Grey River.
A watch and act alert has also been issued for Wongarra.
A recommendation to evacuate Lorne, Allendale and Cumberland River remained in place overnight and relief centres are open at Apollo Bay and Torquay.
Victoria Police officers visited homes in Lorne overnight, urging residents to evacuate the town.
Surf Coast Shire CEO Keith Bailey said almost 100 people sought refuge at Torquay on Friday and 31 spent the night.
Community meetings will be held at Torquay at 10am and Apollo Bay at 2pm.
A cool change on Friday evening brought a difficult wind change for fire crews, but also some cooling rain.
It was still falling over the Otways in the early hours of this morning and the Bureau of Meteorology is expecting up to 30 millimetres on the fire ground.
State response controller Peter Rau said more homes could have been lost.
“This fire is pretty significant in size. We’ve had a lot of firefighters on it,” he said.
“It’s been very traumatic for a number of people. To lose a property at any time is traumatic, but on Christmas Day is something these people will remember forever, unfortunately.
“Our hearts go out to those who have lost properties.
“There could be more, but we need to wait [for] our rapid impact teams to evaluate the area. We won’t know of any further losses until we get in and have a good look around.”