What do you think is the most difficult language to learn and why?

That depends on the languages the learner already knows: on how different or similar the target language is from them.

The American Foreign Service Institute has created a difficulty list based mainly on how many study hours are required to reach competency (not fluency), for a monolingual English speaker.  On their list, the four most difficult languages all belong to other language families and are very different from English. They all use different scripts than the Roman alphabet (which English uses) They are:

1. Arabic - highly inflected.  Many words consist of roots of 2 or 3 consonants over which various patterns of vowels are laid to produce a family of related words. Has grammatical gender (English doesn't;  It has words that represent physical sex, a different concept). Uses an abjad script (no specific letters for vowels but they can be represented in other ways, but often don't need to be written)

2. Mandarin (and all other Chinese languages).  No inflection.  Word order reigns supreme and there are many special constructions. Complex tonal system. Logographic script (characters represent meanings instead of sounds. Many words required at least two characters).  About 4000 are needed for daily literacy.

3. Korean - highly agglutinative and moderately inflected. Honorific forms to represent social relationships among speaker, listener, and those discussed. Uses a different alphabet.  Syllables are written in blocks.  Words are not separated by spaces. Standard Korean has no pitch accent (a simplified tonal system) but some dialects do.

4. Japanese -  highly agglutinative and moderately inflected. Honorific forms to represent social relationships among speaker, listener, and those discussed. Different levels of politeness. Pitch accent system (varies among dialects).  The writing system employs three scripts (a subset of Chinese characters, and two syllabaries - where symbols represent mora, the Japanese idea of a syllable).  

Japanese is considered slightly more difficult than the others, but not enough to increase the number of study hours.

English is difficult for some foreigners, but not all.  There is very little that makes English unique, other than its specific combination of features.  Almost everything, good or bad about English, exists in some other languages as well.  Some foreigners find English fairly easy (especially many fellow Indo-European language speakers). 

The language of love. Why? because my wife NEVER stops giving me a hard time. 

It's different for each person / It's individual. Then the motivation counts and it also depends on people.

Usually the choice among experts is between Northwest Caucasian languages (only 2 or 3 vowels, and up to 84 distinct consonants), Khoisan languages from South Africa (up to 115 distinct click consonants), or Southern Athabaskan languages like Navajo and Apache (notoriously complex verb system, only a few outsiders ever managed to master it).

That's a question of familiarity. It's often said English is the hardest because it is very inconsistent. As soon as you think you've figured out the Latin roots, you're dealing with a German set of rules, then Olde English and on and on. Portuguese is known to have tons of irregular conjugations. A similar problem where as soon as you think you know what's going on, you find out you don't.

What would be a big road block personally is coming from English and Romance base is either having to read characters or learn a tonal language. Mandarin is both. One of the funny quirks is the tone used can mean you say "mud/grass horse" i.e Lama. Or "f---- your mother."

"Hello" in a NY accent means "eat **** and die asshole" to a Californian so I'm familiar with he struggle of tonal languages but still...