Why didn't Dumbledore know who killed his sister?

I don't think he wanted to know. If he knew, it would either confirm he was a killer or his odd brother was. I think he was happier not knowing since that way, there would always be some doubt that it was he that her. That way, he couldn't place all the blame on him or his brother.

And it is thought that Grindelwald knew which brother really killed her and that during later meetings, he told Dumbledore who really killed his sister. If this is in fact true, maybe Dumbledore didn't want anyone else to know, especially Aberforth. If he killed Ariana, then Aberforth may never want to see him again. If Aberforth actually killed her, he may feel so guilty, he might have done something like kill himself.

I think not knowing or knowing and not telling anyone was Dumbledore's way of not placing all the blame on him or his brother. Hope this helps.

I don't think Dumbledore thought of it at that time. We have to realize that Dumbledore had just lost his sister. I'm sure at the point her thought of Priori Incantatem, it was a long while after her death and Dumbledore would have had to waited a long time and he could have ended up waiting forever if he had in fact not killed Ariana.

Also, even if he did that and he did not kill Ariana then he's only left with his own brother and crush. I think the kind of person Dumbledore is he would rather blame himself.

Well because J.K. Rowling wanted to keep it a mystery. But maybe Dumbledore didn't want to know the truth, or didn't think of it at the time.

He didn't want to know. he didn't want to risk that just in case it had been him.

He didn't kill her.