Peter Forrest scored the first century of the series with 104 to carry Australia to 280 for six off their 50 overs, but it proved insufficient as Sri Lanka kept ahead of the run chase and survived a late wobble to seal victory.
Skipper Mahela Jayawardene produced a man-of-the-match performance with 85 off 81 balls and combined with Dinesh Chandimal's 80 from 100 balls to hand Sri Lanka a strong launching pad.
The Australians looked as if they might fight back for a win, taking three wickets in the last five overs, but Thisara Perera (21 not out) and Nuwan Kulasekara (4 not out) got Sri Lanka home with some impressive hitting.
Kulasekara hit the winning runs as Sri Lanka reached 283 for seven off 49.2 overs.
The win puts Sri Lanka top of the triangular tournament on 15 points, one ahead of Australia, with India slipping four points behind the host nation.
India now face a must-win game against Australia in Sydney on Sunday.
Jayawardene looked set for a well-deserved century, the second in the match, before he was stumped by Matthew Wade well out of his crease after being deceived by a beautiful delivery from spinner Xavier Doherty in the 27th over.
"The youngsters are learning a lot from these games. We are top of the table, but we still need to do a lot of hard work," Jayawardene said.
"A couple of them haven't had much experience obviously, but I thought they handled the situation very well.
"They didn't go for those big shots, they just kept it going (with singles). They set it up nicely."
Chandimal was out leg before wicket to Ryan Harris as he walked across the stumps to a straight ball and tried to dink it square, leaving the visitors on 243 for five in the 45th over.
Dan Christian finished the best of Australia's bowlers with three for 53 off eight overs, although Perera clouted him for four and six off successive deliveries in the penultimate over.
"Unfortunately we couldn't get over the line, but full credit to Sri Lanka," Australia skipper Michael Clarke said.
"Our execution with the ball wasn't good. We just have to execute better than we have throughout the whole series.
"Sri Lanka showed tonight that if you hang in, you can take it home."
Earlier, Forrest, slotting comfortably into deposed Ricky Ponting's number three batting spot, smashed 104 off 138 balls.
Forrest hit 10 fours and two sixes to take his tournament tally to 234 runs in four innings since his unexpected selection for the tri-series.
His hundred was not only his first for Australia but also in any one-day match.
He brought up his century by dropping the ball into the off-side and darting for a quick single, before celebrating with fist-pumps.
He combined in a 154-run third-wicket partnership with Clarke (72) after the loss of openers Matthew Wade (5) and David Warner (7) for just 27 runs.
Forrest looked at ease against the Sri Lankan bowling before holing out in the 41st over to deep cover off Angelo Mathews.
Clarke, in his first match back since hamstring-related trouble, struggled throughout with a stiff back in his 79-ball knock with five fours and two sixes.
The Australian skipper was out to a stunning catch by Perera in the 39th over for Mathews' second wicket of the innings.
David Hussey remained unbeaten on 40 off 28 balls and paceman Brett Lee 20 not out with Mike Hussey out five overs from the end of the innings to a crashing yorker from Lasith Malinga for 21 off 14 balls.
Australia, who won the toss, left out fit-again all-rounder Shane Watson and paceman Clint McKay, while the Sri Lankans named an unchanged side.
Copyright AFP (Agence France-Presse), 2012