In British culture "Bugger" has almost no negative connotations anymore. It's a term of endearment if anything. The phrase...

"You daft bugger"

roughly equates to...

"You silly sausage"

(Wait, wait, bad example, maybe American's don't have that one either?) It roughly means...

"You ninny"

(Wait bad axample again!) It has the same meaning as...

"You silly sod"

(Errrm might not know that either?) Ah hah! I got how to explain it. It means the same as...

"You daft apeth"

Hope that's all much clearer now. My work here is done