One of the big "gotchas" in coding against the SharePoint Object Model is not disposing of resources properly. In relatively little time, you can find that your server is cycling itself to free up resources, with no apparent reason. The first thing you should check is your own code.
Roger Lamb has written an excellent, in-depth article called SharePoint 2007 and WSS 3.0 Dispose Patterns by Example about how to fix these issues. When my client had memory leak issues, they first wanted to blame the third party Web Parts that they had installed and they uninstalled them. No joy. After much back and forth with Microsoft (I was out of the loop on much of this), they were pointed to Roger’s post. Making all of the changes that Roger recommended solved the problems.
These weren’t "big deal" Web Parts, either, but they were called frequently. Every time that they were called, they held onto a little bit of memory until everything was brought to its knees. No matter that they were running big iron for their farm; memory is finite, and eventually you will use it all up.
While computing has changed drastically since my early days of writing assembly and FORTRAN, smart use of memory never goes out of style!