I think the matter is contextual.
A comma wouldn't seems ok here:
Of course not.
It may be a matter of personal style, but I also wouldn't put a comma here:
"Are you happy?"
"Of course I'm happy."
Seems weird to add a comma there, and here
Of course I'll come to the party.
As I see it, if we would pause as we say it, a comma should be there. Otherwise, no. Seems like we don't pause if we want to emphasize the "of course".