I think I understand your issue.
From the garbage collector's perspective, there is just a single list of all objects. When making passes, it just starts with known roots and propagates out, until over time, it's touched all live objects. I don't think somehow finalizing the gmMachine's global table is going to help.
What you may need is explicit destruction of your known resources, which you could control.
I haven't looked at how you've implemented custom gmUserObjects to see how straight forward it would be in your situation.
If I recall correctly, here's a couple of things myself and others have done:
1) Instead of storing a C++ ptr or resource directly inside a gmUserObject, store a handle or smart reference which can be nullified.
That way, in script, the gmObject can be alive but C++ object logically dead. This allows the application to control the lifetime of the native object and its dependent resources rather than the gmMachine.
2) When creating gmObjects that represent C++ objects, have C++ own them as a GC root.
On creation, call machine->AddCPPOwnedGMObject(m_userObject); // C Own gmObject
On Nullify, call machine->RemoveCPPOwnedGMObject(m_userObject); // C disown gmObject
If you didn't do something like this, you'd need to store a reference to all C++ owned objects in the global table, or such, just to keep them alive.
Related to (2), there is a templated helper called gmGCRoot<> which can help Add/Remove CPP owned objects.
I recall on shutdown, the code order must be:
gmGCRootManager::Get()->DestroyMachine(m_machine); // For gmGCRoot, call before destructing gmMachine
gmGCRootManager::Destroy(); // For gmGCRoot
I'm trying to recall and describe ways to make use of gmObjects which may finalize (called Destruct in the docs), in undefined order, while allowing the application to control the lifetime of native resources.
Let me know if I'm making any sense. I'm not very current with script issues.