Castiel couldn't be everywhere at once. He couldn't watch his friends at all times; he couldn't protect them when he didn't know what was going on. If they didn't have contact with Magneto, he could at least be sure they weren't being lulled into a false sense of security and friendship--or even just familiarity--only to be killed or turned on whenever it suited Magneto. And Chuck had surely simply been worried about him, because he was a good person, and Mary would reach out to almost anyone. Yes, they were adults, but they were exactly the types of his friends he was trying to protect.

"You made incredible efforts not to speak with anyone you didn't desire to after you murdered Crowley. Don't act as though you couldn't have simply ignored their attempts to contact you."