Subashini Rajasingam also failed to persuade the federal court her husband should be banned from converting their four-year-old son to Islam.
The court rejected the appeal on a technicality but added that she would be able to try again in a few months.
Religious rights are a sensitive issue in Malaysia which is 60% Muslim.
But there are also large Hindu, Christian and Buddhist communities – mainly drawn from the ethnic Indian and Chinese minorities – who fear for their rights.
Malaysian law leaves family matters to Islamic Sharia courts when both parties are Muslim. Non-Muslim matters are dealt with by civil courts.
But there has been a series of cases in recent months where civil courts have appeared to give ground to Sharia courts.
The latest case started when Subashini’s husband – then called Saravanan Thangthoray – announced last year that he was converting to Islam.
He said he wanted to divorce her in a Sharia court, which he would also ask to rule which parent their sons, aged four and two, should live with.
Subashini wanted the matter dealt with by a civil court instead, and asked the federal court to step in.
The court declined to give a ruling, saying she should have waited until three months after her husband’s conversion before making her application. Her lawyers say she will now do so again.
Lawyers are still puzzling over the ramifications of the judgement, which some are calling contradictory.
The judges said that in principle, this case should still be dealt with by the civil courts. But they also said that as a Muslim, the husband had the right to apply to a Sharia court.
They also said that the two types of court have equal status in Malaysia.
“This has created some confusion. We don’t know what impact this will have,” said Subashini’s lawyer K. Shanmuga.