He arranged the kidnapping of Lyla and pretty convincingly looked like he was about to kill Diggle until Thea shot him. That seems like a pretty good reason to lose faith in him, brainwashed or not. Not to mention we, as the viewers, already know that he was willing to kill Diggle once.