one legged dumbbell squat?

In what way would you be wasting a set or two? If you did a couple of extra sets figuring it out, so what? Or, if you spend your entire time that you would have done it figuring it out, so what? Then you know what weight to start with at your next workout. That's part of working out, you're not wasting anything.

Start at a weight that you know is a little lighter than you need for the first workout. Next time you do legs, increase it a bit - even if it is still too light, do the workout. Repeat. After a few workouts, you'll have the weight you need, and along the way you'll have valuable practice getting your form correct before you get to heavier weights.

If you've never done the lift before I would recommend using no weight. Get used to the movement and gradually add weight as you go.

I think you would go to the gym and talk with a trained counselor.