Conservative former energy minister said he cannot support Kings Speech