Real estate experts are heralding residential mortgage REITs as a critical player in any proposal to kick-start private lending as the federal government reduces its role in the housing market.

But the idea is having a tough time gaining traction. A series of worrisome events has created jittery investors, hindering the ability of mortgage REITs to raise capital.

Mortgage REITs make money on the spread between interest rates on short-term debt that they use to buy higher-yielding, long-term mortgage securities. They then pay out the difference in dividends.

Like other sectors, mortgage REITs are challenged by investor anxiety over the European debt crisis. But other events have specifically affected mortgage REITs. These include Congress’ hesitancy to pursue reform of Fannie Mae and Freddie Mac and a Securities and Exchange Commission query into whether the trusts should be exempt from registering as investment companies.

The Obama administration’s latest plan to help underwater homeowners refinance is among the more recent triggers to spook investors. Mortgage prepayments, along with a flattening yield curve, ding mortgage REIT earnings.

Total returns among residential mortgage REITs were slightly negative for 2011 as of Oct. 31, according to the National Association of Real Estate Investment Trusts. Last year, the group generated an average total return of more 21 percent.

Still, government sponsored enterprises Fannie Mae, Freddie Mac and Ginnie Mae and the Federal Reserve hold trillions of dollars in mortgages on their balance sheets, which account for 90 percent or more of the home mortgage market. That’s up from about 50 percent historically.

That level of government support is widely considered unsustainable. Mortgage REITs appear to be an obvious buyer of not only existing loans, but also of new originations once lawmakers get around to reforming the GSEs.

“There’s a seismic change happening in the way that the U.S. finances mortgages and manages mortgage risk over the next five or 10 years,” says Sean Kelleher, chief investment strategist on the fixed-income team at Chicago-based Shays Assets Management. “We’re really opening up a window for new players, and REITs are extremely well-positioned to be the eventual winners.”

Indeed, the U.S. Treasury in February proposed three options for winding down the Fannie Mae and Freddie Mac. Soon after, nine financial firms registered with the SEC to launch new mortgage REITs and raise billions of dollars in IPOs.

Like existing residential mortgage REITs, the IPO candidates generally planned to buy “agency” mortgage-backed securities backed by the GSEs. But most were also looking forward to the return of the private mortgage industry.

To date, only three have completed IPOs: Alexander Mortgage REIT, Apollo Residential Mortgage and American Capital Mortgage Investment Corp.

Nevertheless, Richard Adler, a managing director and co-founder of New York-based investment adviser European Investors, suggests that the conditions are in place for a mortgage REIT boom over the next 10 to 15 years.

Regulations introduced in the 1990s as a result of the savings and loan bust initiated massive growth in Equity REITs, says Adler, who has been investing in REITs for 30 years. Now new mortgage and banking rules taking shape under Basel III, the Dodd-Frank Act and other measures should provide an impetus for mortgage REITs to flourish.

He also sees the vehicles as important income investment options just as Baby Boomers are starting to retire. On average, residential mortgage REITs were paying an annual dividend yield of 15.75 percent at the end of October, according to NAREIT.

“When you change rules you create opportunity,” Adler says, “and mortgage REITs are one way to fill a need for investors while providing a source of mortgage funding.”

Evidence suggests that an appetite for privately funded mortgages exists. Seattle-based Redwood Trust, which buys non-agency jumbo mortgage loans and bundles them for sale, has sold $658 million of the mortgages in two MBS transactions this year. Those followed a $230 million securitization last year.

Finding a supply of mortgages to buy is a challenge, however, and Redwood is establishing relationships with mortgage originators to beef up its pipeline, says Mike McMahon, a managing director with the REIT.

The reduction in the conforming loan limit to $625,000 from $729,500 on Oct. 1 should help a little. The limit caps the size of mortgages that GSEs can buy, and before the crisis it was $417,000.

Now, however, some members of Congress are trying to increase the limit again. Is this yet another setback for mortgage REITs? Brad Case, senior vice president of research and industry information at NAREIT, doesn’t think so.

“There are always discussions about conforming loan limits,” he says. “But there will always be a place for mortgage REITs that buy agency debt and those that buy non-agency debt.”