Well, according top, 385M is for buffer cache, so that can be thought of as "free" memory, since it will be discarded when there is memory pressure. On an OS like Linux, all of your memory is going to be in use soon after you boot up by design (because whatever would be free will go to buffer cache).
The fact that all of your memory is "in use" isn't an issue by itself. However, you are using 631MB of swap. Just speculating, but some process (or processes) may have a large virtual memory footprint due to internal memory leaks where they leave large amounts of dead data in their address space but unfreed. Pages containing data that is dead to the application will eventually get paged out because they haven't been used in a long time. The OS doesn't know that they'll never be used again, so it has to save them in swap if it wants to remove them from RAM. You can try restarting various long-running apps to see if your swap footprint goes down a lot after certain ones terminate.
Also, why is your load average around 2? It doesn't look like it's from heavy CPU usage, so are there processes doing heavy I/O? That could give the system a sluggish feeling. Also, emerald looks like it is using a relatively large amount of memory (almost 250MB resident). Maybe you should turn off 3D desktop effects.