The Dow Jones Industrial Average rose a scant 7.77 points (0.05 percent) to 16,072.54, a record high.
The broad-based S&P 500 slipped 2.28 (0.13 percent) to 1,802.49, while the tech-rich Nasdaq Composite inched up 2.92 (0.07 percent) to 3,994.57. Earlier the Nasdaq topped the 4,000 mark for the first time since the 2000 dot-com bust.
Markets sold off in the last hour of the session in subdued trade at the start of the holiday-shortened week. Markets will be closed Thursday for Thanksgiving Day and have a shortened session Friday.
The main indices opened with modest gains as investors digested news that the United States and five other major powers had struck an interim agreement with Iran over the weekend.
The six-month agreement would be a preliminary step toward resolving the major powers' concerns that major oil producer Iran is developing nuclear weapons, a charge Iran denies.
But stocks lost momentum as the National Association of Realtors reported that US pending home sales slowed for the fifth straight month in October.
And, after the Dow and S&P closed at record highs Friday, markets were taking a breather, said Peter Cardillo of Rockwell Global Capital.
Leading social network stocks fell sharply: Twitter lost 4.7 percent and Facebook 3.1 percent.
Dow member Wal-Mart Stores rose 0.8 percent after the global retail titan named Doug McMillon, a company veteran who heads its international operations, to replace chief executive Mike Duke when Duke retires early next year.
Chip maker Qualcomm fell 0.6 percent after saying Monday that it faces an antitrust probe in China.
Nasdaq heavyweight Apple added 0.8 percent after buying PrimeSense, an Israeli motion-sensing technology company. The financial terms of the transaction were not disclosed.
Struggling department store chain JC Penney jumped 3.6 percent despite being booted from the S&P 500 index, effective the coming Friday. The bump-up came after Imperial Capital raised the company's bond ratings to 'hold' from 'sell', according to The Street.com.
Bond prices rose. The yield on the 10-year US Treasury edged down to 2.73 percent from 2.75 percent Friday, while the 30-year dipped to 3.82 percent from 3.84 percent. Bond prices and yields move inversely.