Is 'g' is truly constant when an object is undergoing free fall?

No it's not a constant; while falling, it increases by the inverse square of the distance relationship; g = g' (R/r)^2 = 9.81 m/s^2 where r = 6173 km is Earth's radius (average).

EX: Assume we start our fall from R = 6193 km, which is 20 km above the surface. The gravity field strength would be about g' = g (r/R)^2 = 9.81*(6173/6193)^2 = 9.75 m/s^2 when we jump out of the balloon's gondola. But hey that's only e = (9.81 - 9.75)/9.81 = 0.006116208 ~ .6 % error if we just go ahead and treat g as a constant 9.81 at that altitude.

And that's why we use 9.81; it's the average for Earth's surface, but it's close enough for most other altitudes. Of course when doing satellites and stuff hundreds of miles up, we have to make the changes in the g value as at those distance from the gravity source (e.g., Earth) the gravity field strength will be significantly reduced.

It is nearly constant. g=9.81m/s^2 is average near Earth's surface. Gravity decreases as the square of the distance so as object free falls gravity gets stronger, but over short distances the change can usually be neglected.