Prime Minister Tony Abbott on Monday refused to comment on the reports, and instead questioned the strength of Labor's commitment to turning back asylum seeker boats after it adopted the policy at the weekend.

It comes as Labor signalled that should the party be elected, asylum seekers would be returned to the country they were fleeing, rather than a transit nation, if their refugee claims were rejected.

There are reports that the group of Vietnamese asylum seekers, which is thought to include two babies, arrived back in Vietnam on Sunday. Australian authorities are believed to have intercepted the group's boat off the coast of Dampier in Western Australia last week.

VOICE spokesman Trung Doan, whose Melbourne-based organisation advocates for Vietnamese people in need of protection, said the group of 46 people arrived in Ho Chi Minh City by plane about 1pm on Sunday.

Quoting sources in Vietnam, he said the asylum seekers were earlier transferred from their boat to an Australian vessel, where they were interviewed by authorities.

Mr Doan said the asylum seekers were then moved to a larger navy ship and taken to an unknown island off the Australian mainland, where they departed by plane on Saturday night.

After landing in Vietnam, he said, the group was transported by road to Vinh Thuan province. Forty-three asylum seekers were allowed to return home but three members of the group have been detained for further questioning, he said.

"There was no explanation. [The three people] were taken … to the provincial centre and that indicates they would be in for serious interrogation," he said.

He said the asylum seekers comprised two extended family groups, mostly Christians, but also Buddhists. He believed one family were experienced boat fishers, and the families pooled their money to buy the boat, which he said left Vietnam on July 2.

There had been speculation the group was fleeing religious persecution, but Mr Doan said he did not believe this was the case.

He said sources in Vietnam reported members of the returned group had lost weight "and looked downtrodden".

"They didn't know anything about Australia's attitude to boat people … They thought Australia, being a civilised country, would treat them better than the Vietnamese authorities," he said.

Fairfax Media asked Immigration Minister Peter Dutton to substantiate the reports. A spokeswoman said the government would not comment on "operational matters".

Mr Abbott said Australia would "act in accordance with Australia's national interests and not to run a shipping news service for people smugglers".

He claimed his government succeeded on border protection because, unlike the former Labor government, "we haven't felt the need to broadcast what government is doing on a moment-by-moment basis. We haven't felt the need to big-note ourselves."

Labor on Saturday resolved to allow boat turn-backs - a decision opposed by key party figures including frontbencher Anthony Albanese.

It also pledged to double Australia's refugee intake over a decade, commit $450 million to the United Nations refugee work in south-east Asia and the Pacific, and lead development of a regional "humanitarian framework".

Mr Abbott claimed Labor was "dangerously divided" on border security "and in many respects almost unrecognisable from the Greens".

He said Labor's national conference suggested that if the party returned to power it would be a "rerun of [former prime minister] Kevin Rudd who, as you know, promised to turn back boats but in fact more than 800 illegal boats arrived on Labor's watch."

Greens senator Sarah Hanson-Young said reports the asylum seekers had been handed to Vietnamese authorities were "extremely alarming".

"The Abbott government must come clean with what it has been done with these men, women and children," she said.

"There are grave fears that those handed back to the Vietnamese authorities will be punished, jailed and further abused as a result of trying to escape."

According to Human Rights Watch, Vietnam's human rights record is "dire in all key areas" and the communist state "suppresses virtually all forms of political dissent, using a broad array of repressive measures" including torture.

On Monday Opposition Leader Bill Shorten indicated a Labor government would turn asylum seekers back to their home countries if they were not deemed to be refugees.

The practice of returning asylum seekers to the country they were fleeing, rather than to a transit country, is a controversial one.

Mr Shorten said Labor would assess if people faced danger or persecution when deciding whether to send them home, adding it would "honour our obligations under the Refugee Convention".

He would not rule out returning asylum seekers to particular countries, saying the circumstances of each asylum seeker would be assessed.

Follow us on Twitter