"I feel that I cannot continue to work with the required professionalism and passion as I have done,," the 65-year-old told a press conference at a Johannesburg sports club.
Baxter was widely unpopular with the South African public, while the media welcomed his decision to leave a job reportedly worth 490,000 rand ($33,500/30,000 euros) a month.
Reacting recently to constant criticism, Baxter said there were "54 million national coaches in South Africa", a reference to the 58.7 million population of the country.
Although Bafana Bafana (The Boys) stunned Cup of Nations hosts Egypt in the round of 16, they made a timid exit against Nigeria in the quarter-finals.
They had been even more unimpressive at the group stage, losing to the Ivory Coast and Morocco and struggling to overcome minnows Namibia.
Previously, Baxter had flopped in an 2018 World Cup qualifying campaign, losing four of five matches as South Africa finished bottom of a group won by Senegal.
The Wolverhampton-born coach had a first stint coaching South Africa with little success a decade ago, failing to take the country to the 2006 World Cup.
He is the ninth coach to quit or be fired since the Cup of Nations ended on July 19 with Algeria defeating Senegal in the final.
Former Dutch star Clarence Seedorf was the highest profile casualty with defending champions Cameroon sacking him following a last-16 loss to Nigeria.
Hosts Egypt fired Mexican Javier Aguirre, Namibia dumped Ricardo Mannetti, Guinea ditched Belgian Paul Put and Tanzania dropped Nigerian Emmanuel Amunike after poor campaigns.
French coaches Herve Renard and Sebastien Desabre left Morocco and Uganda respectively and Sunday Chidzambwa quit Zimbabwe.