This fix is for the vista sp1 update that includes IE 8. Once IE 8 is installed, the folders open in new windows even though the 'open each folder in the same window' option is selected.
This can be corrected by running the IE 8 in administrator mode just once. This will fix the issue. The reason for the issue is still unknown. I'll update it once I get the reason.