What if we forget dreams and forget being a baby for the same reason? What if the problem isn't memories but the lack of an operating system upgrade to retrieve the memories?
Babies are learning like crazy and remember stuff and talking too, but when they are older, most of the memories before some age like four seem inaccessible. I think the development of an "I" or personality is what triggers the ability to recover and use past memories. Before that, memories are just there being an influence.
Dreams are also full of memorable events and long term memory tracks are being laid down. As evidence, consider how a dream is sometimes recalled hours after waking. The dream tracks are presumably all still in place from our youth but only the few we happen to have remembered upon waking can be revisited.
Our personality has less capacity than our mind and hasn't found a way to integrate knowing what we have dreamed with knowing what we have done awake. The dreams are faded daily. If you want to be bothered, you can recover a lot of them and visualize them thereafter. As a young man, I collected several thousand dreams but it was a lot of work and they generally didn't make much sense or seem important.
What kind of personality will be capable of remembering all the pretending of the night and its feelings and actors and stay sane while dealing with related feelings and some of the same actors by day? It's not yet known but don't rule out the possibility. With the right upgrade, the last twenty years of your dreaming may suddenly pop into active memory.
(Is that a RAM upgrade or a REM upgrade :-) )