Australia's Barty, the French Open champion who was competing for the first time since a fourth-round exit at Wimbledon, battled back from an early break to take the opening set, but she was unable to do the same in the next two sets as 20-year-old Kenin triumphed 6-7 (5/7), 6-3, 6-4.
"It just wasn't there today," Barty admitted. "At times I was missing by big margins, which is not something that I'm very used to or comfortable with."
The defeat puts Barty's number one ranking in jeopardy, with both Japan's Naomi Osaka and Czech Karolina Pliskova eyeing the top spot.
Second-seeded Osaka will open her campaign on Wednesday against Germany's Tatjana Maria, and if the Japanese star wins there will be a new number one next week.
Osaka is the front-runner to return to the summit, but Pliskova could have a chance if she makes the semi-finals.
"I couldn't tell from you a bar of soap what's going on with the rankings," Barty said. "It's not something that I worry about or focus on. Regardless of whether it's number 10, it's number 50, 200 or it's number one, it's just a number next to your name."
Pliskova next faces US qualifier Alison Riske, who saved three match points on the way to a 4-6, 6-2, 7-6 (9/7) victory over Greece's Maria Sakkari.
Croatia's Donna Vekic also saved three match points on the way to a 3-6, 7-6 (7/5), 7-6 (7/5) win over American Madison Keys.
There was no coming back, however, for seventh-seeded American Sloane Stephens, who fell 6-2, 7-5 to Czech qualifier Marie Bouzkova.
Bouzkova, ranked 91st in the world, notched her first win over a top-50 player.
"I honestly still can't believe it," said Bouzkova, who broke Stephens five times -- taking full advantage of the former US Open champion's 48 unforced errors.
Stephens will now try to regroup with the US Open fast approaching, as will Barty -- who insisted after her defeat "it's not panic stations."
- Andreescu beats Bouchard -
============================
Kenin, who has climbed the rankings this season with wins in Charleston and Mallorca and a runner-up finish in Acapulco, notched her first victory over a top-five player in her fourth attempt.
She needed four match points to seal the victory, with Barty saving three on her own serve in the ninth game of the final set before Kenin served it out.
Trailing 3-5 after an early break in the first set, Barty won three straight games before Kenin held to force the tiebreaker.
Barty opened a 5-1 lead in the decider and pocketed the set on her third set point.
Kenin raced to a 4-0 lead in the second set only for Barty to regain two service breaks. Kenin then broke again to lead 5-3 and held at love to force the third.
Teenager Bianca Andreescu won her first match since Roland Garros, rallying to beat former world number five Eugenie Bouchard 4-6, 6-1, 6-4 in an all-Canadian first-round clash that lit up the night session.
Andreescu, 19, shot to prominence with her triumph at Indian Wells in March, but she has battled a nagging shoulder injury since and was playing her first match since the French Open.
"It's been a rollercoaster," Andreescu said. "But I've been preparing really well for the last couple weeks, and that preparation really showed tonight."