The EU's main audit body says agricultural subsidies - the biggest item in the EU budget - often go to people who do little or no farming.
A new report by the European Court of Auditors complains of deficiencies in the Single Payment Scheme (SPS), which distributed about 29bn euros (£26bn) of subsidies in 2009.
It says payments "have become divorced from current farming conditions".
The EU is considering how to reform its Common Agricultural Policy (CAP).
The auditors have made various recommendations to the European Commission aimed at improving the SPS - the biggest area of spending in the CAP.
They say the SPS ought to direct aid to "active" farmers and provide more balanced funding so that a small number of big landowners no longer get the lion's share.
They also call for clearer definitions of land eligible for subsidies and of farming activities.
The report complains that the 17 EU countries applying the SPS use about 20 different variants of the payment scheme, making it too complex.
The SPS does not operate in 10 EU countries, which joined the EU in 2004 and 2007. All 10, except Cyprus, are former communist countries and they use a different system of farm support, called SAPS.
The auditors say the SPS has encouraged farmers to respond better to market demand and has benefited EU agriculture as a whole.
But they say the way the scheme's beneficiaries were defined "permitted persons or entities not, or only marginally, engaged in an agricultural activity to benefit from SPS payments".
In some cases landowners have carried on receiving the payments even though their land is worked by tenant farmers who do not get the subsidy.
In the UK the auditors found some individual beneficiaries receiving up to 1m euros annually or even more in SPS aid without having any agricultural activity on their land.
The report also highlights examples of non-agricultural land qualifying for SPS payments in France, Italy and Spain.
The European Commission has said EU farm spending should no longer be based on previous subsidy levels for farmers.
But the commission believes subsidies are still needed to protect Europe's food supplies and rural diversity. The proposals are contained in an EU blueprint for farming beyond 2013.