IN ESA animation the probe rosetta was shown going in triangular shape curve. Was the orbit really triangular If so how this and why?

I noted the same thing, I don't actually know the answer to this, so I can just speculate, but I have some ideas.

From an animation I have seen it seams that that almost triangular "orbit" was in the initial phase of the Rosetta encounter with the comet, the comet was not in the center of the triangle(comet was out of the "orbit" plane) and the relative distance was relatively large and then the probe come closer and the orbit gets a much more usual curved-line shape.

This suggests 2 ideas, the first is that the probe might was out of comet gravitational sphere of influence(i don't know the scale of the animation, comet's SOI is about 600km), and therefore it was not orbiting the comet but was still orbiting the sun on an orbit very similar to the comet's one, and those adjustments were just to take a look to the comet from different point of views before being in orbit(why? don't know exactly but maybe to fit the entire comet into a single shot from the narrow angle camera or for being always in the enlighted side). Or it was in the SOI but in an open trajectory(iperbolic) and was basically going back and forth on iperbolic trajectory, not actually always the same iperbolic trajectory but progressively less eccentric and with lower perigee(of course maneuvres like this would require a HUGE deltaV on a planet or big moon and this is why they are so unusual, but may be affordable with such a small gravity), this could be useful to better estimate the gravity constant to design the later orbital maneuvers or just to get closer with a safer step by step strategy instead of a single risky maneuver

somehow similarly to a capsule that perform a rendezvous and docking to the space station, no capsule ever docked with just a single engine burn and perfectly hit its dock, they keep doing smaller and smaller adjustments (with small trusters), At a diferent scale with the sun instead of earth, the comet instead the space station, and the probe in place of the capsule this was a sort of docking maneuvre (altough the dynamic is not the same since the space station has not it's gravitational sphere of influence you can orbit in)

Can you link to the animation you've seen? It probably looks like that because the 3-D orbit is being presented in 2-D.

You mean like in this graphic?

http://sci.esa.int/rosetta/54901-rosetta-trajector...

Because 67P's gravity is so weak, Rosetta is mainly doing 'station-keeping' rather than orbiting, though it can do orbits at 20 km and 30 km.