Worst case is that you don't know how to run a software team and you end up spinning your wheels for ages and then have to onboard a Dev team and convince them they want to clean up someone elses tech debt. Thus making everything cost 10x what management initially thought.
If the agency devs are so mismanaged that the project is a disaster, why on earth would you then hire in house devs instead of reverting to the external platform?
Agencies serve as a good middle ground between buy it (not working for OP at the moment) and going through a hiring process for an entire engineering org you’re not convinced you’ll need. Agency not working? Terminating the project is so much easier than firing the 6 devs you just got through HR.
This is probably the last thing I would suggest.