Advice on making games without engines?

If I just have C++ and say SDL or Raylib, how do I structure game code to keep it scalable (ie not a huge mess when I add more levels, items, mechanics, etc)? I have been able to make very simple stuff but the moment I try to add to it, it always gets out of hand and I can't really refactor it without starting fresh.