Just because facilities wouldn't necessarily be top notch (according to a league's norm), who cares if the atmosphere is charged and the fans are excited? The NFL can be hosted by the home of the Crimson Tide.
If there is less seating for the event, the ticket price is higher. So, if a team is in town only 4 times, well... people will pay. It's more of an event in that town than just another week.
Look at some of the cost of building and operating these sports structures. Article links below:
|Metalist Stadium in the Ukraine|
Cowboys and more listed
Falcons Football and Cincinnati mention of cost:
Hockey arenas and NBA
For the love, grassrootsy:
Article: Baseball and the American City. An examination of public financing and stadium construction in American professional sports.
See more expansion and relocation talk for football, baseball, basketball, hockey and soccer on my blog (starting in Feb 2014, see table of contents/archive): Major League/Pro Expansion <CLICK> http://prosportsexpansion.blogspot.com/
And please feel free to contribute your two cents with mine.