The more memory you can assign for your YaCy peer, the best it will behave. I think you can safely and should at least run it with 1500MB.
What precise version of YaCy do you run (you will find it in /Status.html, YaCy version line)?
I had similar memory performance problems not so long ago when running with memory below 1024MB (see http://mantis.tokeek.de/view.php?id=626
). For me the problem mainly occured when storing large documents metadata from remote peers.
If you run last YaCy versions, you can now "Limit size of indexed remote results" in /ConfigPortal.html page. When limiting to 1000KB, it worked for my peer and allowed me to run YaCy with 600MB of memory.
You may also want to remove too large documents from your index : see IndexDeletion_p.html page, "Delete by Solr Query"
For example, such a query will delete documents with more than 10000 embedded links : inboundlinkscount_i:[10000 TO *] AND inboundlinks_urlstub_sxt:[* TO *]