The number of customs checks at main UK airports has dropped because staff have been moved to work on passport control, the Immigration Service Union claims.
The union told the BBC that passengers were being waved through without extra checks, leaving the UK open to a higher risk of arms and drug smuggling.
It said a strike on Thursday, over pensions, would worsen the situation.
The government says it is committed to maintaining border security by deploying staff flexibly.
Long queues at passport control in Heathrow and Stansted airports have been a source of embarrassment to the government in the run-up to the Olympics.
There have been reports for several weeks about the delays, with passengers arriving at Stansted on Friday saying they were delayed by several hours. Airport bosses have called the situation "unacceptable".
Firearms and drugs
Both the home secretary and immigration minister have pledged to deal with the delays.
But with customs staff being moved to help with passport checks, the union's deputy general secretary, Lucy Moreton, said Heathrow staff "are having to reduce all other activity, and that includes the secondary activity of checking for drugs, firearms, and such like."
Random searches of people "have all but ceased", she added.
"We would be fools if we believed that the organised crime gangs behind this aren't aware and haven't been aware of this for week," Ms Moreton said.
"It's important that the border must be secure... both from individuals who seek to exploit it as people, and also from people who would seek to exploit it for drugs, for firearms and for other items."
Customs staff from Luton, Manchester and Stansted were also being brought into Heathrow to help out, she said, meaning those three airports were now suffering from passport checking delays themselves.
"We need a 100% secure border, not patches moved around in response to media hype," Ms Moreton said.
Movement of staff between duties was normal, but the excessive queues at Heathrow had meant more staff than usual had been deployed to immigration control.
Ms Moreton said that although the Home Office claimed to be deploying staff flexibly, further flexibility was needed.
There are also reports of Olympic workers entering the UK unchecked.
An unnamed border official has described an incident last week involving two students from Pakistan with Olympic credentials that he felt might pose a security risk, but entered unchallenged.
Ms Moreton said: "I'm not at all surprised. With the pressure of the queues in front of you... there is a lot of pressure just to let these people go."
Thousands of civil servants and other public sector workers are set to go on strike on Thursday.
The ISU, representing 4,500 workers spread over ports and airports across the UK and abroad, is one of three unions with staff in the UK Border Force, which carries out customs functions and immigration checks. Border staff in the Public and Commercial Services union are also set to join the action.
The dispute with the government is over plans to increase the retirement age for public service employees, linking it to the state retirement age.
Ms Moreton said: "Nobody wants to go on strike but we are simply unable to get the Cabinet Office to engage with us."