What is considered copyright  infringement ?

Creating any unauthorized derivative work (like a hand-made copy) is a violation of the copyright owners' exclusive rights, subject to various exemptions.  Merely doing it "for fun" is not an exemption. Doing it for "study or research" may start you in the direction of the exemption for "fair use".  However, once YOU created an unauthorized copy, you become responsible for its status as unlawful, if you give it away (or when you die), and could be sued for infringement.  

Statutory damages are available for many works in the USA, depending upon when their copyright was registered.  That could mean you're sued for $150,000 for each work that you copied without permission, if your copy was illegally created or used.

On the other hand, taking a photograph of a work in public is not an infringement, even though it is a derivative work, as long as your "use" is legal (e.g., fair use, educational use in a non-profit classroom, advertising exhibition or sale of the original, etc), as provided by law.  When you get sued, you have the burden of proving either an actual license to make the copy or a statutory exemption for your creation.

Your mileage may vary, if you're sued in countries outside of the USA, if that's your concern. 

Also worth noting: if the copyright of the original has expired, or it never had any copyright, then you cannot possibly infringe rights that no longer exist.

Different countries have different legal systems, so let me focus exclusively on American law for the sake of clarity.  17 U.S. Code § 106 gives copyright holders certain exclusive rights which includes the right "to reproduce the copyrighted work in copies or phonorecords" and "to prepare derivative works based upon the copyrighted work."  17 U.S. Code § 501 defines an "infringer" as "Anyone who violates any of the exclusive rights of the copyright owner as provided by sections 106 through 122 or of the author as provided in section 106A(a), or who imports copies or phonorecords into the United States in violation of section 602."

To rephrase without the legal jargon, you are a copyright infringer merely by reproducing a copyrighted work without the owner's permission, even if you never sell, distribute or profit from it.  Technically you can be sued for this reason alone.  Although it's unlikely the court will determine that there are any actual damages involved, you could still face statutory damages (which are penalties the law assigns simply for infringing copyright).  These run up to $150,000 per infringement, though in this scenario, it's more likely they will be near the lower than the higher end of the spectrum.

Practically speaking, it's unlikely that this scenario will lead to a lawsuit because it looks really bad from a public relations standpoint to sue someone for creating unauthorized fan art which they never sell or profit from.  But it isn't impossible.  Perhaps with the recent establishment of the copyright small claims court in the U.S., we will see some lawsuits of this kind because this kind of court makes it easier for small creators to sue infringers, and there are a few creators out there who are VERY protective of their works.

The original artist would have good legal cause if it wasn't designated as a copy.  But most likely would not pursue action because suing someone else costs money and without damages to collect, doing so would be a waste of time and money.

In order to "sue" you must first prove some kind of damage resulting from your work being copied.  A personal copy made for personal use...even personal acclaim...does not "damage" the original painter.

Attempting to sell the work or taking credit for it yourself DOES damage the artist's right to profit from his own work (in terms of both money and reputation)

Museums are often full of student and hobbyist painters who openly copy works by other artists.  This is traditional and accepted.

There's nothing an artist can legally do if you just copy his/her work and hang it in your home.  There's nothing he/she can do to prevent your friends and family from praising your work and complimenting you.

If you sell your copies or use them for personal gain (example: you advertise to do paintings on commission, then include your copies in your portfolio as copies, but to show off the quality of your technique),  you could expect to get a "cease & desist" letter from that artist's representatives.

No.   Not if the copy made it clear this is not the original. 

He could argue there is no difference between his work and a person who takes a photo of the original art. 

If the person was not going to sell it and so was not attempting to make money from it,  what would be the loss for the original artist?