When do i take finch nest out of cage?

My baby finches started to come out of nest. Its been 3-4 days now ..when should i take the nest out of the cage? I also want to know if i can leave babies in with parents once they feed themselves. Thank you