Affordable housing ‘neglected’ in CBD