Can an American, English speaking person attend colleges in Japan?

Yes. If you want to go to a regular Japanese University, you would have to prove your proficiency and pass the entrance exam.

There are universities like Temple, APU, and possibly Sophia that are known for taking international students because many of their classes are in English.

I know what you are talking about. I've seen those kids over and over and everywhere on the campus. We called them, "Bunches" because they all bunched up with same Japanese buddies all the time. I've never seen them trying to interact with other students from other countries or Americans. I think they are shy and too afraid that no one will understand their English. Most of the Japanese 'bunches' who talk and play only with Japanese were the exchange students who only stay for a few months to study English. Their Japanese school was the sister college to mine in U.S. Other Japanese students I saw came to persuade their degrees in U.S. were more outgoing and made friends with Americans and other students from other counties. It was unfortunate that they didn't enjoy the time in U.S. with other people. I must wonder why the school even want to spend the money to be wasted. They attend the classes and go back to dorm room and play with other 'bunches' and drink in the evening. It's a shame. I hated these people because I was also a Japanese student who came to study (I didn't come from the same school as these kids, though.), and saw these people doing the exactly the same thing they do in Japan. The other Japanese students who came individually weren't interact with these "Japanese bunches" also, because it is just a waste of time. There are so many things you can learn from each other. Why you want to waste the perfect opportunity? I don't get it.