How do we know that time is affected near black holes?

It's been demonstrated that time slows down in a gravitational field and the stronger that gravity is, the more time is slowed down.  It stops completely once the escape velocity exceeds that of light.

"Like do we have any evidence of how time appears to slow down? "

I assume you have a phone with gps?  Chances are it's accurate to a couple of feet or so but if it didn't correct for time dilation the accuracy would be miles rather than feet.

Dude, we can measure the effect at different altitudes on Earth, so we know it will happen near a black hole.

Because it is a solution of Einstein’s general relativity which has been verified by every experiment ever done.

The time dilation factor - for a idealized non-rotating black hole with respect to a far away observer - is √(1-Rs/R) where Rs is the event horizon radius and R is the radial distance from the black hole. You can plug in various values of R to see the time dilation factor for a given event horizon. Note the special case where R = Rs where time is zero from the perspective of the outside observer.

For someone in the gravitational field of a black hole, time proceeds normally for them - even at the event horizon - in a local region around them.

Time is "affected" in the vicinity of any massive object. The higher the mass, the more that time is affected. It can actually be measured in the vicinity of the Earth. An object with several times the mass of the sun crammed into a volume the size of the Earth will therefore have an even stronger effect upon time.