Bencic's 7-5, 6-4 triumph, her WTA-best sixth over a top-five foe this year, ensured 21-year-old Japanese star Osaka will fall from the world number one spot next week.
"The challenge cannot be bigger against Naomi," Bencic said. "I had to be at the top of my game and I'm really pleased how well I handled my nerves at the end."
Bencic, 22, matched her deepest career Grand Slam run from the 2014 US Open with her third victory of the year over Osaka, having also won at Indian Wells and Madrid.
"She played pretty clean. She had a specific plan she wanted to execute," Osaka said. "I guess just being aggressive."
Australia's Ashleigh Barty, the reigning French Open champion, will move atop the rankings next Monday after the loss by Osaka, who defends her Australian Open title in January.
"I can look at this and be very disappointed and mad, but I'm not mad about it," Osaka said. "Of course I'd like to reach higher rounds. That's definitely what I'll aim for in Australia."
Vekic, the 23rd seed from Croatia, saved a match point in the second set and made her first Slam quarter-final by rallying past German 26th seed Julia Goerges 6-7 (5/7), 7-5, 6-3.
"I don't even know how I won this match," Vekic said. "She was serving for the match. She had match point. I just kept fighting and believing I could win. It feels pretty amazing."
Bencic owns a 2-1 career edge over Vekic from two 2014 wins but Vekic won the most recent meeting with her friend and frequent practice partner in the third round of this year's French Open.
"It's going to be a tough match for sure," Vekic said. "She's one of the best players this year."
Osaka's exit means there will be four different women's Grand Slam winners in a season for the third consecutive year, a first in the Open era.
- 'Unbelievable moment' -
Reigning Australian Open champion Osaka struggled from the start on the same Arthur Ashe Stadium court where she consoled tearful US teen Coco Gauff just two days earlier.
"It was just an unbelievable moment for tennis," Bencic said of Osaka helping Gauff cope with her sadness. "Coco's story is unbelievable but what Naomi did is what a true champion would do."
Osaka took a pain-killer to ease nagging knee issues but nothing could help her overcome Bencic.
"I didn't move well. I was always flat-footed. She was aggressive," Osaka said. "The knee was a little bit annoying in the movement aspect but that was something I should have overcome."
Osaka double faulted to surrender a break in the opening game of the match, but leveled at 2-2 when Bencic netted a backhand.
Bencic swatted a backhand down-the-line winner for a break to seize a 6-5 lead then held on a service winner to capture the first set.
Osaka double faulted away a break to hand Vekic a 3-2 lead and the Swiss never faced a break point in the second set, finishing off the victory with a forehand winner to end matters after 87 minutes.
"I was taking the serve early, trying to anticipate, because she has so much power," Bencic said. "I'm trying to play like chess and make it tactical on the court."
Bencic hit 29 winners with only 12 unforced errors to Osaka's 26 winners and 21 unforced errors.
Goerges won the tie-breaker with consecutive forehand winners, then broke Vekic to open the second set and fired four consecutive aces for a 2-0 lead.
But the German was broken serving for the match, squandering a match point in the 10th game, and Vekic broke again when Goerges hit a forehand wide to force a third set.
"I was just trying to get a return in the court," Vekic said. "She was serving amazing. I was feeling confident in the rallies and I knew if I could get that return in I would have an advantage."
US upstarts play seeds in later matches as 116th-ranked qualifier Taylor Townsend faces Canadian teen 15th seed Bianca Andreescu and Belgian 25th seed Elise Mertens meets 141st-ranked wildcard Kristie Ahn.