As you are on ADSL, it all depends on the quality of the wiring in your house and between your house and the DSLAM in the telco's CO. Double check that you have the filters in the right place (on all phones and not on the modem) and make sure your modem is not near anything like a CRT or a fluorescent light or even right next to a computer as all those can put out interference that limit the speeds the modem can run at.
Try looking up your modem on DSL Reports and see if there is a management page that you can use to dump out the speeds and line quality it is getting - that will tell you what is going on.
Definately look at the wiring in your house. Worse comes to worse, run a new phone line from your phone box outside to your DSL modem and see if your speeds improve. Or, if you want an easy way to do it, run the modem directly into the box. Cheap and easy test.