We had a similar problem recently which was only noticeable against CRM Online/CRM Hosted. For us the solution was to use REST instead of SOAP in our calls to CRM from Silverlight. We saw reductions in response times of around 90%.
I've seen similar problems and solutions with PHP SOAP calls as well. Just looks like SOAP is really slow in some sites, and REST has a lot less overheads.
There are a lot of posts on this subject as well suggesting to use the compression capabilities of IIS (gzip from memory) to speed up the download time of the Silverlight app.
Graham Hill
Metisc