Sometimes compilation with gcc/g++ can take up a huge amount of memory, especially with high optimization options. Not sure if it's a bug or just the nature of compilation but it seems to be a common problem. You could have tried compiling with a lower optimization setting or increasing swap space. Now that QB64 is compiled and working you should be fine to reduce to 2GB if you prefer.