Pick 4 points in the given squares; what's probability they are the vertices of a convex quadrilateral?

This solution is incomplete, but here is how I see it. For a convex quadrilateral one of the interior angles must be a reflex angle. Suppose it is vertex A.

Let there be B(r, s) and D(t, u), and let BD cut the axes at E(0, v) and F(w, 0). As I interpret the conditions, B and D have uniform distributions on their respective unit squares, which means r, s, t, and u are independent and uniformly distributed on their respective unit intervals. The axis intercepts can be derived from those four random variables.

v = (st - ru)/(t - r)

w = (st - ru)/(s - u)

Interior angle A is a reflex angle if line BD crosses quadrant I and A lies in ∆EOF.

P(BD cuts quadrant I) P(A is in ∆EOF | BD cuts quadrant I)

= (1/2)(area of ∆EOF)

= vw/4

= (st - ru)(-r + s + t - u) / [4(t - r)(s - u)]

That would be the integrand for a quadruple integral with respect to the four variables across their respective intervals. The result would be the probability of a reflex angle at A. A quadrilateral can have no more than one reflex angle, so reflex angles at this and the other three vertices are mutually exclusive events. That means we could get the probability of a convex quadrilateral by adding the probabilities, which is equivalent to multiplying this probability by four.

Probability of convex quadrilateral

= ∫[-1,0] ∫[0,1] ∫[0,1] ∫[-1,0] (st - ru)(-r + s + t - u) / [(t - r)(s - u)] dr ds dt du

I could not complete that integration on my best day, and my limited available calculus software balked at it. I was able to use The Geometer's Sketchpad to set up a randomized simulation. With 10,000 iterations, it returned about 0.09 as the experimental probability.

It seems like we used to get more creative questions like this one. I would be interested in seeing that solution you mentioned. I see no harm in answering your own question.

I was thinking it would be similar to the square region case for Sylvester's Four Point Problem (25/36), but you have the added restriction that each point is in a separate quadrant.

I'll see if I have some time to play with this, but ultimately I hope you'll share your approach.