Many people in the education sector are "frightened" to speak out against Education Secretary Mike Russell, Labour has claimed.
Mr Russell was also accused of trying to "bully and intimidate" college staff at Holyrood.
It emerged at the weekend that Mr Russell called on the chairman of Glasgow's Stow College, Kirk Ramsay, to consider his position after a private conversation about controversial college reforms was recorded and distributed.
Labour education spokesman Hugh Henry said education staff are "frightened to speak out" against Mr Russell, calling him a "bully" and citing the dispute with Mr Ramsay as evidence.
Mr Russell replied that most of his meetings with the college sector are positive and said Mr Ramsay was wrong to record their meeting. He said he has "no power to demand any individual's resignation", as ministers' power of direction over the college sector was removed by the previous Labour Scottish Executive.
Mr Russell said: "The Scottish Government believes that the college sector, like any other, needs to be led and governed by people of the highest quality and standards. Chief amongst their attributes must always be mutual trust and respect. The Scottish Government expects all those who lead and govern our colleges to conduct themselves in a way that was consistent with that approach."
Mr Henry suggested it may be appropriate to record meetings for accuracy and argued that it may be "in the public interest to publicise what the Cabinet Secretary said or how he behaves at meetings".
He said: "The issue is his reaction to this event I've been contacted by FE (further education) staff who believe that the Cabinet Secretary is behaving inappropriately and that he is attempting to bully and intimidate."
Mr Russell rejected the claim, arguing that "the issue is that in a meeting of chairs and principals a recording was made with a surreptitious device". Mr Russell did not take up Mr Henry's invitation to publicly apologise to Mr Ramsay.
Mr Russell added: "There is no dispute between the Cabinet Secretary, the Government, or anybody else, between myself and Stow College, the staff, students or board. This is a matter of the conduct of one individual. That is where the dispute occurred."