Is it very hard for learners who learn Japanese to become able to understand verbal Japanese?

No, spoken Japanese isn't all that hard. It's very different to English, which obviously makes things more difficult as you have to start from scratch, so to speak, but the language itself is not that hard, and it's made up of not very many sounds compared to English. This is the reason Japanese people struggle to properly learn spoken English: English has a lot of sounds which simply don't exist in Japanese, and also English ends most syllables on a consonant, whereas that doesn't happen in Japanese, and it's something most Japanese people really struggle with.

The reason Japanese is usually rated as a difficult language to learn is based mostly on the writing system, with it's 1000-odd characters. 

The U.S. state department ranks languages

based on how much time it takes - on average - for a native English speaker to become fluent in that language.

They have experience educating thousands upon thousands of people in this regard

so their data and statistics are quite sound.

Japanese is ranked the most difficult language in their list.

Now: that includes both speaking and writing

so I can't answer your question specifically with regard to speaking

BUT

typically it's much easier

a WHOLE lot easier

to learn how to read a foreign language

than it is to learn how to understand a native speaking that same foreign language.

(Why: a native tends to speak very quickly

from the student's point of view

and

a native tends to run words together, without clear pauses between words

which thing the typical foreign language student has little experience with)

That's true even if the speaker is using "standard" accenting.

If the native speaker has an accent, it's even more difficult to understand them...sometimes MUCH more difficult.