this book was written by who/whom?

WHOM....

The words to, for, by etc must be followed by 'whom'.

Who wrote this book? ( 'who' is the subject, 'book' is the object).

This book was written by whom? ( whom is the object, 'book' is the subject)

Who wrote this book?

This is a better way of asking the question as it is is far more concise and active.

As for the matter of who vs whom, a good way to remember which one to use is to answer the question with he or him using the same form. If you use the word he in your answer, then the proper choice would be who in the question. If him is the proper choice, then whom should be in the question.

He = who

Him = whom

This book was written by whom?

This book was written by him.

Who wrote this book?

He wrote this book.

expositots bible