In the figure,a square of edge length 17 cm is formed by 4 spheres of masses m1 = 4.7g,m2 = 2.9g,m3 = 0.8g,&m4 = 4.7g.What am I doing wrong?

Assuming m1 and m4 are on opposite corners so they cancel out which is why you didn't work out those forces.

In your calculation for r where do you get 0.1 m from?

The distance from center to corner is r = sqrt( (0.17 m)^2 + (0.17 m)^2) / 2 = 0.1202 m approx.

The formula for gravitational force between two point masses says you need to SQUARE THE DISTANCE.

It also says you need to multiply BOTH masses together.

And for example 2.9 g = 0.0029 kg and not 0.029 kg... be careful with your powers of 10

In light of that

F25 = G(m2)(m5) / r^2 = (6.67×10^-11 Nm^2/kg^2)(0.0029 kg)(0.0029 kg) / ((0.17 m)^2 / 2) = 3.88×10^-14 N

m3 is 3.625 times less massive than m2, so F35 = F25 / 3.625 = 1.071×10^-14 N

Resultant is F25 - F35 = F25 (1 - 1/3.625) = 2.811×10^-14 N

cos45° and sin45° are necessarily EQUAL because those angles form an isosceles right triangle with two legs of the SAME length so the i- and j-components must be equal.

2.811×10^-14 N * cos45° or sin45° = 1.988×10^-14 N

So resultant on m5 is

F = (1.99e-14 i + 1.99e-14 j) N <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

rounded to 3 sig digits.

Assuming m2 is kinda between the i and j directions.

If the length of side is indeed 0.1 m, then all the forces are increased by a factor (0.17 / 0.1)^2 = 2.89 so multiply answer by that.