"He's a complete fraud," Cara Loening, director of Sign Language Education and Development in Cape Town told AFP.
"He wasn't even doing anything, There was not one sign there. Nothing. He was literally flapping his arms around."
The interpreter, who translated eulogies including those of US President Barack Obama and Mandela's grandchildren, looked as if he was "trying to swat a few flies away from his face and his head".
"The deaf community in South Africa are completely outraged and nobody knows who he is," said Loening.
"We can't find a name or anything. The organisations who have accredited interpreters do not know him at all."
The government said it had launched an investigation following the allegations, the results of which would be made public.
Minister in the Presidency Collins Chabane said at a news conference the government was "looking into this matter" but would be unable to conclude its investigations on Wednesday due to other pressing demands ahead of Mandela's funeral on Sunday.
Attended by nearly 100 sitting and former heads of state or government, the speeches at Mandela's memorial on Tuesday were supposed to be interpreted into sign language for deaf viewers.
But the on-stage interpreter's signing appeared at odds with that of the public broadcaster's signer, who was shown in a small insert box on SABC television screens.
The White House meanwhile referred all questions on the matter to the South African government but said it would be regrettable if the incident overshadowed Obama's "very powerful remarks" at the memorial service.
"It would be a shame if a distraction about an individual who was on stage in any way detracted from the importance of that event and the importance of president Mandela's legacy," deputy spokesman Josh Earnest told reporters in Washington.
Interpreter 'drew complaints before'
South Africa's Translators' Institute said they already had reservations about the interpreter, who has previously signed at the ruling ANC party's events, including its centenary anniversary last year.
"We suspect there was something underhand about him," the institute's chairman Johan Blaauw told the domestic SAPA news agency. "How else was he working on such an important occasion when he did not seem to know what he was doing?
"I'm afraid this thing has left the whole of South Africa with egg on its face."
He said there had been complaints over his signing at past ANC events, yet the ruling party took no action.
Delphin Hlungwane, a spokeswoman for the Deaf Federation of South Africa, said the man picked for the job was "just gesturing in the air".
"He didn't interpret at all, he had zero percent accuracy," she said.
"International people were watching it as well and they all said he was not using any kind of sign language. They all said they don't understand him," said Hlungwane.
Loening said her organisation was getting emails from around the world "wondering what on earth this man was doing there".
"It's a real embarrassment. It's complete disrespect for the deaf community and for what Nelson Mandela stood for and the support which he gave toward the deaf community."
While sign language semantics may differ from country to country, the basic grammar remained similar, said Blaauw.
"The common grammar can be recognised across the various sign languages," said Blaauw. "It's been the response across the world that he wasn't using the language but just making funny gestures."