This is a problem that lots of writers have, particularly in the SFF genres: how much is too much when it comes to building a world?
I’ve been lost in the weeds myself recently, chasing down threads that have no real bearing on overarching narratives. When I find them, I decide to compose a rich backstory, something with heft and weight, that has no bearing to the larger narrative.
Good to know the history of a religious schism that only ever had one adherent–do they matter?
And that, I think, is a good rule to follow when writing your story. World building is important–less so than internal consistency, but that’s another topic–but it’s just detail. If there’s nothing compelling happening around those details, you’re just writing a mechanism to deliver a lore wiki.
Story First. Character First. Plot First. Let the details enhance the story, rather than crowd it out. Only use as much as the story demands, and the bigger moments of world building will have that much more impact.