How best to save for building a new home?

Here’s my situation. I am a single mom, my kiddo will start school (kindergarten) likely starting in Fall 2022 (she’s got one of those weird summer birthdays, so it may even be a 2023 start.) I’m in a great school district, but HATE my home ($1500/mo payment, includes taxes, homeowner’s insurance, and HOA fees.)

In 2022, a better chunk of my student loans will be paid off (frees up about $500/mo) and in early 2023 all of my federal loans will be paid off (frees up an extra $700/mo, for a total of extra $1200/mo). I have private loans that won’t be paid off until 2027 ($550/mo payment). By summer 2021, I will have no other consumer debt aside from my student loans and my mortgage payment.

My big, main, long-term goal is to build my own home that will be the final home I own. I want everything in it to be absolutely what I want, which is why I want to build - I’m not leaving until they take me out feet first. I work from home full-time, LOVE my job (very stable - make about $170k/yr), like the area where I live (cost of living is rather low), and really have no intention of quitting it/retiring anytime soon. In speaking with a builder in the area, “all in” on the home I want would be about $500k with a 10% contingency built into the budget (this is everything - land, permits, home, painting, utilities, cabinets, etc. etc.)

I’m wondering what the best plan would be to start saving for this home? Is this realistic in a reasonable time frame? What amount should I aim for to have in hand before even starting to think about building? Do I focus on wiping out the private loan balance first, or do I put the extra money that is now in my budget in index funds every month?

Of course the builders are aggressive in that I don’t have to have much down to start building, but I really REALLY really don’t think it would even be smart to start without at least 20% on hand, but the new home build is what’s throwing me off, as I’m unsure how PMI and such work into any loan for a new build.

Any advice would be helpful. Is my dream too big? How aggressively should I be saving? What should I theoretically aim for to have on hand if I want to build a $500k home?