Toxic rhetoric around benefit cuts has stirred up hatred against disabled people, Tanni Grey-Thompson has said, as campaigners warned the government’s planned welfare changes were “brutal and reckless”.

Lady Grey-Thompson, a Paralympic champion and crossbench peer, said she had been contacted by disabled people saying they had been shouted at in the street by passersby telling them they were “going to get their benefits cut”.

“There’s some really horrible rhetoric around at the moment,” she said. “It’s not a great time for disabled people. And the worry is that this doesn’t do the right things to get people into work. It’s whether it could push people who are currently just about surviving into greater poverty or more need.”

The TV presenter and disability advocate Sophie Morgan, who is paraplegic, said she saw no justification for the cuts and said the “scapegoating of disabled people is not only toxic, it could be really fatal”.

“If history has taught us anything, these cuts will have the opposite effect of what the government seems to think they will have. It will push people out of work, it will cause more poverty,” she said.

“I don’t think there’s a single disabled person in the UK who won’t be affected by this. But I think everyone should be worried – if the government can treat people with the highest needs in our community this way, what else could they possibly do? It’s a very worrying question.”

She added: “There is this terrible misconception that disabled people just take, but actually by taking from us, you prevent us from being able to give – we want to be able to participate in life equally the same as anyone else, and that includes going to work.”

The work and pensions secretary, Liz Kendall, on Tuesday announced benefit changes including tightening the eligibility rules for personal independence payments (Pip) and reviewing the assessment process, as well as cuts to health-related universal credit.

Charities and disability advocate groups criticised the approach, saying the changes would “drive more disabled people into poverty”.

Mikey Erhardt, a campaigner at Disability Rights UK, said: “After months of rumours, media speculation and spin, it is clear these reforms were not about supporting disabled people into work, but instead simply about making brutal and reckless cuts amounting to £5bn.

“Rising claims for Pip reflect not a problem with disabled people but rather reflect successive governments’ failure to do even the bare minimum to create a more equitable society.”

Paul Kissack, the chief executive of the Joseph Rowntree Foundation, said: “A government that came to office pledging to end the moral scar of food bank use clearly should not be taking steps that could leave disabled people at greater risk of needing to use one.”

He said “enormous cuts” risked undermining some of the positive reforms announced by the government, such as the “right to try guarantee” to help people back into work.

The disabled activist Dermot Devlin said he was almost in tears as he watched Kendall’s speech in the House of Commons on TV.

“I had to turn it off because it was too much. The fact that they were putting it across as good news when in fact anyone who is disabled knows it’s anything but good news, it’s absolutely devastating for our community,” he said.

“I don’t know if it will affect me, but we don’t know what the parameters are yet. It might not affect me now but it could next year. I’m just a bit lost for words.”

Devlin, who has mucopolysaccharidosis type IV, also known as morquio syndrome, said he relied on Pip to help him cover the costs of the powered vehicle he uses to get around, and the mask he wears to help with his breathing while he sleeps.

“Public opinion is getting nasty towards disabled people. But Pip is needed for survival,” he said. “You take Pip away and it’s just going to push so many disabled people, including myself, deeper and deeper into poverty.

“I’ve never felt more pushed away from being a member of society.”