What can I do to help him follow directions?

My boyfriend has a serious problem following directions. He listens to the instructions and then it seems he skews them in his brain or something. For example: this morning he had to walk 4 blocks to work. We live on East 38th Street and his job today was on East 38th Street as well. I told him to go out of the house and walk down the block, the name of the first street he would come to, etc. Somehow he wound up 10 blocks in the other direction. I know if it had been me, I would have known something was wrong within 2 blocks (when I saw the numbers going down) and turned around or asked for help. He became very fustrated (he always does when things aren't going his way) and blamed the neighborhood for getting him lost. So I guess my question is really has several aspects: Why is he having so much trouble, how can I get him to acknowledge that it's him, not the universe/neighborhood/etc, and what can I do to help?