Why do people think third rails are in the middle?

The Live conductor rails can be in practically any position, depending upon the whims of the original engineers who built the systems in the first place, although a 'third' rail on one side of the running lines in the most common.

The 'third' rail is NOT in the middle in London: on the London underground system, this is in fact a FOURTH rail, used in combination with a conventional outside third rail. This dates back to the earliest days of the system, when electrical engineering was less sophisticated, and by providing an outside 'live' rail, and a centre rail for the return current, the traction currents could be kept away from the running rails, which carried the low voltage currents for track-circuiting, i.e., to operate the signals. In fact, on sections of the line where earth leakage was a problem, the outside rail carried a positive voltage of 300 volts, and the centre rail a corresponding negative voltage. Thus, there was a potential difference between the two rails of 600 volts, the normal voltage used by the trains.

However, the original Central London Railway (now the Central Line) was built with slightly smaller 'tube' tunnels than those later used, and until these were enlarged in 1940, the live rail had to be in the centre, with return through the running lines. The Liverpool Overhead Railway was another system that used a centre live rail originally.

On the Paris Metro some of the lines are operated by the so-called 'Pneu' trains, that have conventional rubber-tyred wheels that run on concrete strips outside the conventional rails. They have horizontally-mounted rubber tyred wheels that bear on vertical steel rails on either side, to provide guidance. These steel rails also provide the electric current to power the trains, so this is a third and fourth rail system - one live and one return, either side of the running rails.

I have no idea what " ... that experiment in France" refers to!

There are also differences in the live rails themselves. The normal system is for the 'shoes' of the train to bear on the top surface of the rail to collect the current. However, just from British practice, the old Liverpool and Southport line used the higher than normal dc voltage of 1200 volts. The third rail was encased in wood on the outside and the top for safety, and the collector shoes bore against the inside edge. The Docklands Light Railway in London uses a suspended light-weight aluminium live rail encased in plastic top and sides, and the collector shoes rub along the exposed bottom of the rail.

As a result of falling power lines blocking streets after the blizzard of 1888, New York City outlawed the stringing of overhead wires. When electric trolley cars were introduced, as they could not have a pole going up to a wire, they got their power from an underground line placed in a slot between the running rails, just as a cable car hooks up to a moving cable in a slot. Washington DC also used this system downtown.

It's probably influenced by O Gauge trains, whose power supply rail is in the middle. I believe this design feature was developed by Joshua Lionel Cowen.