How do scientists know if a certain fossil is on our ancestry or on a different path?

The simple answer is that we don't know if a given fossil is a direct ancestor of a given species. By determining its age and looking at its features, we can place it roughly in the right place in a lineage, but it's always possible that a fossil is an example of an evolutionary offshoot that had recently diverged from the main line.

In actuality it's their best guess, based on years of experience in dealing with similar fossils and modern skeletons.

They don't. According evolution scientists everything originated from the "big bang," where a singularity suddenly exploded and all the atoms spread out, and evolved over billions of years. So there for everything is our ancestor. Silly right? I challenge you to think about this: Where did the first atom come from? The singularity had to be made up of atoms. Didn't it? Where did they come from? How and why did they explode? How did life come to be? Did it simply wish itself into being? Was it created by a higher power (God)? Why did it evolve? Why did this evolution suddenly stop? Personally i think God is the answer. He is the answer to everything (this question, the world, everything). Pray to him and he will give you guidance.

The form of it.