Leaving the EU would mean "difficult and lengthy negotiations" that could take years to complete, peers say.
The Lords EU Committee said determining the rights of two million UK nationals living in the EU would be a "complex and daunting" part of exit talks.
While they do not recommend a vote either way, peers say trade deals between the EU and non-EU states take between four and nine years on average.
Leave campaigners argue a UK-EU Treaty should be wrapped up in two years.
A separate report from Oxford University's Migration Observatory, also published on Wednesday, suggests curbs on in-work benefits for EU citizens negotiated by David Cameron are unlikely to result in a large reduction in migration to the UK.
It said a "large majority" of recent EU migrants were not claiming benefits of any kind.
If the UK voted to leave in the referendum on 23 June, it would not mean an immediate exit from the EU.
The prime minister could trigger Article 50 of the Lisbon Treaty, which provides for a two-year process for negotiating the terms of withdrawal and a departing state's future relations with the EU.
But there is some debate as to whether this is the best method for leaving the EU and whether it should happen immediately after the vote.
The cross-party committee warned that negotiations for withdrawal and to establish a new relationship with the EU would take longer than the two years allowed under Article 50.
The committee, which took evidence from two EU law experts, said one of the most complex parts of the talks would be establishing the rights of UK citizens resident in other member states to access health care, schooling and employment.
The negotiations, it added, would also have to deal with the reciprocal rights of EU nationals living in the UK.
"We don't take a view on whether the UK should leave the EU or not," said Lord Boswell, the former Conservative minister who chairs the committee.
"But it is clear that if that's what people decide, withdrawal would mean difficult and lengthy negotiations."
"It's not possible to predict exactly how long it would take, but comparable international trade deals have taken on average between four and nine years."
Peers also say the process of reviewing and disentangling the UK from EU law would take years to complete - with the government having to assess which laws it wished to keep.
Vote Leave, the official group campaigning for an EU exit, said the UK faced a choice between "handing more money and power to Brussels or to take back control".
"It [the report] also torpedoes the claims that the EU wouldn't do a trade deal with us after we vote leave - it's in everyone's interest on all sides to strike a deal after the UK takes back the power over trade."
UKIP said the committee was made up of Europhiles and its findings were "partial nonsense".
Remain campaigners argue there is no guarantee that expats living in the EU would be able to stay in the event of Brexit.