Hi Shefali. No, I have not solved the problem. Rather than beat my head against a brick wall I took the obvious better course of action -- we simply went back to using embedded media player and dumped Silverlight.
With all of Microsoft's obvious push to move folks to Silverlight this seems like one of the MOST fundamental issues to solve or provide an explanation for.
If Windows Media Server on Windows Server 2003 cannot deliver Silverlight WMVs at least as quickly as embedded Media Player, something is wrong.
Perhaps it is my lack of knowledge. Even if so, I have been able to obtain any advice on this. That's a pity as I put a great deal of time into coding our sites to use Silverlight, encouraged our users to downlaod, and then had to go back to Media Player due to the slow response of Silverlight on Windows Media Server.