The Country Fire Authority (CFA) confirmed 35 homes had been lost in Wye River, and 18 at Separation Creek on Friday.
Hundreds of firefighters and dozens of tankers and aircraft have been battling the blaze, which started last weekend but flared up in hot, blustery conditions.
The fire, which has so far burned more than 2,000 hectares, is not yet under control.
An emergency warning is still in place for Wye River, Separation Creek, Kennett River and Grey River.
A watch and act alert has also been issued for Wongarra.
A recommendation to evacuate Lorne, Allendale and Cumberland River remained in place overnight and relief centres are open at Apollo Bay and Torquay.
Victoria Police officers visited homes in Lorne overnight, urging residents to evacuate the town.
Surf Coast Shire CEO Keith Bailey said almost 100 people sought refuge at Torquay on Friday and 31 spent the night.
Community meetings will be held at Torquay at 10am and Apollo Bay at 2pm.
A cool change on Friday evening brought a difficult wind change for fire crews, but also some cooling rain.
It was still falling over the Otways in the early hours of this morning and the Bureau of Meteorology is expecting up to 30 millimetres on the fire ground.