International leaders have hailed the election of Hassan Sheikh Mohamud as a "great step forward" for Somalia.
Who is Hassan Sheikh Mohamud?
However, Islamist militants said the election was organised by the "enemies of Somalia".
Somalia has suffered 20 years of war.
Accepting his defeat, former President Ahmed said it was the first fair election in Mogadishu for 42 years - since Mohamed Siad Barre took over in 1969.
President Mohamud, a moderate Islamist academic, was sworn in immediately after it was announced that he had beaten Mr Ahmed by 190 to 79 votes in a ballot of newly elected MPs.
"I promise that Somalia reclaims its place in the world community as of today - and to do that, we must ensure that we move forward," he said in his victory speech.
The UN envoy to Somalia Augustine Mahiga warned there was still much to do. Mr Mahiga said the Somali people had "taken a great step forward on the path to peace and prosperity".
EU foreign policy chief Catherine Ashton has called up Somalia's new leader to congratulate him.
However, the al-Qaeda-linked group al-Shabab still controls many southern and central parts of the country, and has staged frequent suicide attacks in the capital since it was driven out of Mogadishu last year by African Union troops and pro-government forces.
"We will only recognise a process run by Somalis and not manipulated by Ethiopia, Burundi, Kenya, Uganda, Djibouti and the West," al-Shabab spokesman Sheikh Ali Mohamud Rage is quoted to have said.
"Hassan is operating under a constitution drafted by the enemies of Somalia, who have a nefarious geopolitical agenda, especially neighbouring countries."
Hassan Sheikh Mohamud, 56, is a civic activist who founded a university in Mogadishu and has worked for several national and international peace and development organisations.
He has links to al-Islah, Somalia's branch of the Muslim Brotherhood, which is strongly opposed to al-Shabab.
He stayed in Mogadishu throughout the years of conflict, unlike many intellectuals, and he has previously held a post in government.
Like former President Ahmed, he is from the Hawiye clan - one of Somalia's biggest.
President Mohamud could represent a different kind of future for the country because he is not associated with the violence and corruption of the past.
Nevertheless, he faces massive challenges on multiple fronts. Firstly, he will have to deal with the powerful politicians who lost the elections; then he has to try to reunite a country torn apart by two decades of civil conflict, much of which is controlled by al-Shabab.
Since the overthrow of President Siad Barre in 1991, Somalia has seen clan-based warlords, Islamist militants and its neighbours all battling for control.