At what point are employees responsible for their own behavior and work ethic?

I'm a Custodial Supervisor at a university. I directly supervise 20 people. Over the last two years my fellow supervisors and I have been stripped of more and more authority. I make sure all of my part-time custodians are properly trained and I make sure they have the equipment and supplied they need to do their job. Lately I have three employees who just don't want to pull their weight. My boss says that their unwillingness to work is my fault because I haven't given them good enough reasons to buy into their job. my other 17 employees are all hard workers as am I. We aren't perfect and we have our off days but we do our best. Is it really my fault that they don't work hard? I was taught that you do the job you were trained to do with little or no complaining.