Mr Brown, a shareholder in Scottish side Raith Rovers, urged owners and managers to consider the supporters.
Manchester United fans demonstrated against the Glazer family ownership recently and Liverpool and Portsmouth have also come under scrutiny.
"The management of clubs must look seriously at their responsibilities to their supporters," said Mr Brown.
Liverpool are thought to be £245m in debt after Americans Tom Hicks and George Gillett took over in 2007, while Pompey face a winding-up petition from HM Revenue & Customs at the High Court.
At his monthly Downing Street news conference, Mr Brown added: "It is a matter that the clubs themselves have got to be concerned about.
"Of course, in many cases there are very simple ways that they can deal with these problems. In other cases, football clubs don't have the income that is necessary to deal with the leverage that they have.
"But this is an issue and it's an issue football clubs are facing and it's a worry to supporters. These clubs have high levels of income from the supporters but the debt levels have been too high."
In 2005, when he was Chancellor, Mr Brown participated in the buy-out of Raith Rovers' previous owners by a local consortium.
Source: BBC Sport
Source: BBC Sport