GameMonkey Script

GameMonkey Script Forums
It is currently Tue Mar 19, 2019 6:51 pm

All times are UTC




Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Yielding From C Code
PostPosted: Sat Apr 10, 2010 6:10 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
We're considering moving to Game Monkey from the micro-thread based system we've been using. Our "scripts" are all written in C++ and loaded as DLL's or PRX's. We want to move to a scripting language and really like the structure of GameMonkey, but from my understanding of how the VM is run, it's only possible to yield or sleep from within script code.

Am I correct in this?
If I am correct, has anyone looked into wrapping the GameMonkey threads into microthreads?
One step further, if no one has done this, any ideas on the feasibility/complexity?

We really want an interpreted script, but want to be able to continue using our script support routines that yield internally. We have a WalkToPoint for instance that scripts are able to call. Internally it runs in the scripts microthread, wakes once a game loop, does some work and yields. It returns to the script some number of game loops later when it has either finished or hit some fail condition.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Sat Apr 10, 2010 8:33 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
Also, I read an old post about Threading (after posting my comment), but I don't think it quite addresses what I'm talking about, though it did also use a WalkToPoint as an example.

With our current system, every game entity runs its scripts within a microthread. The engine routine WalkToPoint is able to call yield directly, which saves off its registers and stack, and in the next gameloop it picks up where it left off. Its stack is saved, so all of its local variables are saved, and it's able to call deeper into other functions which may themselves yield. The system doesn't care if the yield happens in engine code or in script code. I haven't jumped completely into the GameMonkey source yet, but seeing how it ran on the PS3 with no modifications (which was really a pleasant surprise by the way, integrating GameMonkey into our engine couldn't have been easier), I have to think its threading is just keeps the virtual machine state.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Sun Apr 11, 2010 2:35 am 
Offline

Joined: Mon Dec 15, 2003 1:38 pm
Posts: 708
Hi YourUncleBob, and welcome to the forum :)

I think I understand what you are asking and trying to do, so the answer should be 'yes'.
The cooperative routine actions like yield, block, sleep can be set or triggered from native code instead of within script.
Most of the functionality is available for immediate use, so you can for example implement WalkTo and have that native bound function block the calling script thread until the walk has completed (or failed).

Any native bound function could also return GM_SYS_YIELD to halt execution of that script thread for a script frame, however I'm pretty sure that is not what you desire. I'm also pretty sure you aren't talking about making a preemptive yeilding system as that would introduce a world of pain by making script threading unpredictable.

Certainly one of the main features of GM script is to use the script to perform what would otherwise require native code to save the stack and longjmp back to resume later (or similar implementation). What GM does not do is manage CPU time spent by script threads. It is up to the programmer to use efficient logic or appropriate thread operations to manage CPU usage (if it is a bottleneck at all).

As far as managing gmThreads from native code, it is possible to do such things arbitrarily, but the functionality may not be ready for you to 'just use'. The usual way is to have script call native bound functions which then perform thread related activity, either normally, or explicitly. I am still thinking that when you say native code 'yield and sleep', you really mean 'block' and resume later as yielding and sleeping actions themselves are not real useful. Other people have modified GM to do things like set priorities on gmThreads so they are processed with different regularity, to manage larger numbers of different type threads. Perhaps that is also an option to consider.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Sun Apr 11, 2010 4:49 am 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
Greg wrote:
Hi YourUncleBob, and welcome to the forum :)
...
I am still thinking that when you say native code 'yield and sleep', you really mean 'block' and resume later as yielding and sleeping actions themselves are not real useful. Other people have modified GM to do things like set priorities on gmThreads so they are processed with different regularity, to manage larger numbers of different type threads. Perhaps that is also an option to consider.

Thanks for the welcome and the quick response.

I'm still missing something here.

I mean yield or sleep here exactly as they are used in GameMonkey code. If I yield() in GameMonkey code, it will resume right after the yield on the next GameMonkey machine execute. I want to be able to do this from native code as well. So, the GameMonkey script running in GameMonkey thread A calls native routine WalkToPoint. WalkToPoint does some work and calls gmMachine::yield() (causing thread A to yield and the machine to move onto the next thread). On the next GameMonkey machine execute I'd want thread A to resume in the middle of the native WalkToPoint. Some number of gameloops later WalkToPoint would eventually return to the script that called it.

If your yield() is saving off stacks and registers and longjmping (or implementing microthreads or fibers in some other way), I'd expect it to work in either script or native code. My guess though was that it wasn't doing that but that it was operating each GameMonkey thread in its own memory space with its own virtual stack, program counter and locals.

I don't understand exactly what you were suggesting. It seems something along the lines of WalkToPoint could cause GameMonkey thread A to block until some later time. But WalkToPoint would still need to return immediately wouldn't it? It was called from script code. And then something in the engine would be responsible for unblocking thread A later? That would make our script support routines a lot more complex than they currently are.

Our current system works by running each script thread in a microthread. Both the native code and the scripting code can call yield() and get the same behavior. The next game loop, the script will pickup on the line of code after the yield() whether it was in a script routine or a native routine. This setup allows the native code to be as straightforward and sequential as the script code.

We could restructure things so that all of our script accessible routines return immediately and all yielding happens in GameMonkey code, but I'd love to be able to have the freedom to yield from anywhere, and I'd love to not rewrite all of our script support routines (other than by changing our MicroThread::sleep() calls to GameMonkey::yield() calls).

We're committed to switching to an interpreted scripting language, and GameMonkey looks really close to allowing us to continue writing the same style code we're used to, if we can just figure this piece out. If it's possible to do with the current system, that would be fantastic. If it isn't we'd consider diving into the GameMonkey threading system and rebuilding it on top of MicroThreads if you think that would be feasible. We're just starting out with this and are trying to figure out if this sort of thing would be worth pursuing before we spend a ton of time digging into the innards.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Sun Apr 11, 2010 6:41 am 
Offline

Joined: Mon Dec 15, 2003 1:38 pm
Posts: 708
Okay, I think I understand some more. Things do not work as you describe, however all may not be lost.

The GM virtual machine is 100% virtual in that the stack, heap, threads, variables and objects are all logical constructs of the scripting language (script compiler and runtimes). They do not relate at all to native code except via the interface classes and functions (gmThread, gmVariable, gmMachine etc.) They do not make use of OS / CPU threads or C++ / C# fibres / micro fibres / longjump() or the like.

So GM won't help your native code act like thread style scripting. GM becomes your thread style scripting. What is nice about GM is that within the script code, it does not matter whether objects represent native or script data or functions. They appear to be the same thing. You may end up with native code calling script and script code calling native, and layers of mixed script and native on the real stack. HOWEVER, when the current native code has finished executing, the stack will always end up in native code ONLY. This does not usually represent a problem, though there are some pathological cases than can cause confusion.

The behavior you describe with WalkTo() cannot be done with native code yielding and resuming. Instead the behavior logic would have to be implemented entirely in script (just the behavior mind you, not all supporting code!). It is the script that can readily yield or block to resume later using a (undesirably inefficient) polling system or (more desirable) triggered event.

Just to clarify some more. An implementation for WalkTo(), would have the behavior script call WalkTo(). WalkTo itself can be either script or native code. WalkTo() may be native but either way, it would block, causing native code to exit as a normal function and continue, BUT script code would stop at the call. Later, while native code in the game loop or script code in other script threads executes the 'walk' logic, a signal will trigger the original script thread to resume.

Rather than me keep talking about this, please have a look at the fantastic 3rd GM tutorial by downgraded:
http://www.gamedev.net/reference/articl ... le2666.asp

These other forum threads may be worth reading also:
Another question about threading viewtopic.php?f=6&t=101
New user, so some questions viewtopic.php?f=5&t=243

Please let me know if I am making any sense.

Edit: Added some clarity to possible WalkTo() implementation.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Sun Apr 11, 2010 6:43 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
Greg wrote:
Rather than me keep talking about this, please have a look at the fantastic 3rd GM tutorial by downgraded:
http://www.gamedev.net/reference/articl ... le2666.asp

Thanks, that's the tutorial that drew me to GM in the first place. It's also the first test code I ran in our game after hooking up GM. Two threads printing messages to our debug window was a snap. It's exactly the style code we use for our scripting. I completely get how the cooperative threading works in script, and think it's fantastic. I'm also hoping to get that same flexibility into native support routines.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Sun Apr 11, 2010 8:25 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
I did a bit of debugging, and unless I'm missing something, it looks like a straightforward addition. I'll give it a shot this week.

Something along the lines of:
Add a needsMicroThread flag to gmFunctionEntry and gmFunctionObject
Add an activeMicrothread pointer to gmThread
In PushStackFrame, if the m_cFunction about to be called needs a microthread, run it in one and set the activeMicrothread pointer
The microthread will return GM_SYS_YIELD,GM_SYS_BLOCK,GM_SYS_SLEEP or GM_SYS_OK
If anything but GM_SYS_OK, the microthread still needs to run during the next Execute
When the microthread terminates (it will return GM_OK), zero out the activeMicrothread flag and continue script execution
When a thread executes, if it has an acitve microthread, switch to it instead of running the next script command

With this setup, just the native routines that can yield(), sleep() or block() need to run in a microthread. Native calls to a nativeYield(), nativeSleep() or nativeBlock() will suspend the microthread and return the proper GM flags to the gmThread so that the normal GM threading system will be responsible for waking the thread.

I believe you'll end up with the best of both worlds. Only microthread overhead for those native routines that need it, and the flexibility to use all of GM's threading system from native code.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Mon Apr 12, 2010 5:22 pm 
Offline

Joined: Fri Jan 14, 2005 2:28 am
Posts: 439
That sounds pretty complex for the example you have given. Am I missing something? You can do asyncronous thread yields/blocks/etc from native functions.

I do similar scripting.

Code:
// this function is asycnronous
if ( ai.MoveToPoint( somePosition ) != PATH.SUCCESS ) {
    ai.Say("Can't path to point");
}


You just have to return the yield or block code from your MoveToPoint function. In my case I do the equivalent of this

Code:
// this function is asycnronous
ai.MoveToPoint( somePosition );
if ( block(PATH.SUCCESS,PATH.FAILED) != PATH.SUCCESS ) {
    ai.Say("Can't path to point");
}


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Mon Apr 12, 2010 7:52 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
DrEvil-OB wrote:
That sounds pretty complex for the example you have given. Am I missing something?

I hope I'm missing something. I'd love for it to be more straightforward.

As I understand it, in your example:
-Script calls MoveToPoint
-MoveToPoint sets some internal engine flags so that the engine can carry out the actual MoveToPoint work over successive frames. Every entity using this routine needs to store its own state somewhere. Different such routines need to store different bits of state information (a FollowCurve for instance).
-MoveToPoint returns immediately to script
-The script blocks waiting for a signal from the engine
-The next GM thread executes
-The game engine carries out the MoveToPoint work over multiple game loops
-When it is done, the engine signals the blocked script to resume it

What I want to do is:
-Script calls MoveToPoint
-MoveToPoint runs in a MicroThread, does some work and Yields(). All of MoveToPoint's locals are preserved, so we don't need any special storage structures or state machines for this routine to run steps in successive game loops.
-The next GM thread executes
-The next game loop when this GM Thread fires, MoveToPoint does some more work and Yields()
-Eventually, MoveToPoint finishes and the script resumes. The script didn't need to do anything special for MoveToPoint to have taken multiple frames, it just called it.

The benefit of my proposed method is that MoveToPoint becomes much simpler to write, and the scripts become a bit simpler to write. We have many script support routines that work in this way, vs. one scripting engine, so I'd prefer the complexity go into the scripting engine. I may be coming at this a bit backwards. We aren't writing our script support systems from scratch to work with GameMonkey, they already exist having been used for several years with our old system.

Am I wrong about how your example functions? Is there a simpler way to go?


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Mon Apr 12, 2010 8:46 pm 
Offline

Joined: Fri Jan 14, 2005 2:28 am
Posts: 439
YourUncleBob wrote:
As I understand it, in your example:
-Script calls MoveToPoint
-MoveToPoint sets some internal engine flags so that the engine can carry out the actual MoveToPoint work over successive frames. Every entity using this routine needs to store its own state somewhere. Different such routines need to store different bits of state information (a FollowCurve for instance).
-MoveToPoint returns immediately to script
-The script blocks waiting for a signal from the engine
-The next GM thread executes
-The game engine carries out the MoveToPoint work over multiple game loops
-When it is done, the engine signals the blocked script to resume it


Mostly right, except MoveToPoint doesn't return immediately to script( in the first example ). If you call block separately then yes.

YourUncleBob wrote:
What I want to do is:
-Script calls MoveToPoint
-MoveToPoint runs in a MicroThread, does some work and Yields(). All of MoveToPoint's locals are preserved, so we don't need any special storage structures or state machines for this routine to run steps in successive game loops.
-The next GM thread executes
-The next game loop when this GM Thread fires, MoveToPoint does some more work and Yields()
-Eventually, MoveToPoint finishes and the script resumes. The script didn't need to do anything special for MoveToPoint to have taken multiple frames, it just called it.

The benefit of my proposed method is that MoveToPoint becomes much simpler to write, and the scripts become a bit simpler to write. We have many script support routines that work in this way, vs. one scripting engine, so I'd prefer the complexity go into the scripting engine. I may be coming at this a bit backwards. We aren't writing our script support systems from scratch to work with GameMonkey, they already exist having been used for several years with our old system.

Am I wrong about how your example functions? Is there a simpler way to go?


Are you talking about an OS microthread(fibers)? Or are you talking about it running in a separate GM thread?

In my example the script already doesn't need to do anything special to call MoveToPoint. It just calls it as if it was a syncronous function and the return value is used to indicate success or failure.

Generally your AI system already has some state machine or behavior tree or something that can do things like navigate the AI to a location, so this would just use that system. Sounds like you don't have this, what do you have on the native side then? In terms of your AI functionality?

There's no way around running logic on every successive game loop until the action completes or fails. If you have a proper AI system that would just be part of their AI update calls.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Mon Apr 12, 2010 9:27 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
DrEvil-OB wrote:
Are you talking about an OS microthread(fibers)? Or are you talking about it running in a separate GM thread?

Good old Games Gems 2 Micro-Threads. Our proper AI system is pretty much one or more behavior MicroThreads per entity. Generally a high-level behavior that monitors big-picture items, and a low-level behavior that carries out the mundane like walking to a point. The high-level behavior is able to switch the state of the low-level behavior if something more important comes along. It's a state machine, but a state is a Micro-Threaded routine with its own local variables. Most of the states are written in script code (which for us is currently C++ code, it's just compiled separately from the engine and loaded as a dll/prx), with the support routines being written in engine code. An entity's Behavior can yield in either script code or engine support code.

With GameMonkey in the mix, what I've been considering is switching that to one or more GM threads per entity and creating all of the state routines in GM, leaving all the support routines in engine.

DrEvil-OB wrote:
Mostly right, except MoveToPoint doesn't return immediately to script( in the first example ).

OK, that I don't get. Both routines were tagged as asycnronous, I guess the first is sycnronous, but I don't get how that would work. If the actual walking to the point is going to take many frames, don't you have to use the second example?


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Mon Apr 12, 2010 11:00 pm 
Offline

Joined: Fri Jan 14, 2005 2:28 am
Posts: 439
The 2nd example is an exploded view of what the first one does internally. The first example has a block on those events built into the call, while the 2nd example has the block separately. Sorry for the confusion.

Microthreads for high level behavior makes sense to me, but I don't understand why you'd run them for low level behavior. In my AI system( behavior tree to be specific ), I have a 'FollowPath' low level state that receives commands from script or native side high level behaviors. Script tells the low level behaviors to move to a point, and it blocks and waits for it to complete. There is no reason for script or another fiber to continue running.

I never felt much need to use fibers for low level routines, as they are generally meant to do incremental work each frame. It sounds to me like it complicates the process.

I suppose that using a microthread for you move behavior would basically be something like this(pseudocode)?

Code:
while( ! ReachedPoint() )
{
    RunToNextPoint(); // updates the next point, with obstacle avoidance, etc
}


Which isn't much different than a path following state that gets an update call each frame.

In my system, low level states always exist, and essentially act as support logic for high level states. A high level state can tell the AI go to somewhere(FollowPath state), while aiming at something(Aimer State), and since these low level states always run in parallel to any active high level state, it keeps things very simple, and since they are always around, the FollowPath state is free to continue running checks even after the AI has reached a point for such functionality such as tethering the AI to the destination area in case he can get pushed around.

Can you elaborate on what you get from putting your low level states onto fibers as well ?


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Tue Apr 13, 2010 3:53 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
DrEvil-OB wrote:
Can you elaborate on what you get from putting your low level states onto fibers as well ?

I see 2 big benefits. The code is sequential, straightforward code, and you have access to persistent local variables.

Before using MicroThreads, we had a lot of code that was essentially a mini state machine. Something like a walk to point would start doing the walk, then either set an active function for the game to use next loop to monitor the walk, or set a state for the game to use in a big switch statement to continue the walk, etc. With a microthread, WalkToPoint can exist entirely on its own. It doesn't need to set any cryptic state flags, or be broken into steps. Whenever it has done what it needs to do in a game loop (look at a nav mesh, avoid other characters, whatever), it just yields and picks up next game loop where it left off.

Local variables persist while a MicroThread lives. Before we used this method, we had a general AI character with a bunch of different internal variables that meant different things to different states. Depending on the behavior, you had to track things like curve to follow, point to walk to, entity to attack, current nav mesh triangle, etc. As we wrote new behaviors, we kept having to either cram more and more variables into the entity classes, or reuse existing variables with non-obvious names. With a behavior running in a MicroThread, we just use local variables and never need to change the base entity class.

The same as a GM thread, between execution steps, all a MicroThread costs is a bit of memory to keep its stack saved. As long as you don't have large amounts of local variables, and don't Yield() from a deep recursion level, the saved stack size can be really small. It's not as if these things are consuming any processor resources when they aren't doing anything.

We indeed have code very similar to what you wrote, the only addition being local variables and a Yield().

Code:
while( ! ReachedPoint(pointToReach) )
{
    RunToNextPoint(pointToReach); // updates the next point, with obstacle avoidance, etc
    MicroThread::Yield();
}

Another example going back to PS2 days would be our memory card code. Before MicroThreads, we had this cumbersome, hard to follow, hard to debug state machine stepping through memory card operations. With MicroThreads, it became much simpler. There were still error cases to handle, but the complexity of the code went down significantly.

I'd say, for pretty much any of the reasons you'd pick GameMonkey's threading system for scripting code, you'd pick MicroThreads for game code. They function almost identically. I'd compare how you'd write the guts of WalkToPoint in GameMonkey, vs. how you'd write it in C without microthreads. I want the flexibility to write it in either place in the same cooperative threading style.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Tue Apr 13, 2010 4:25 pm 
Offline

Joined: Fri Jan 14, 2005 2:28 am
Posts: 439
Very cool. Thanks for the clarification.

So you are wanting to know how to set up your function, ie "WalkToPoint" so that it can function as a native side microthread or a script side asyncronous function? In either case, control would not be returned until it completes right?

That seems pretty simple to do.

Basically if it's a native side function you need to associate the gm thread Id with the microthread that gets created when the script calls the function

Code:
// WalkToPoint microthread
while( ! ReachedPoint(pointToReach) )
{
    if ( !RunToNextPoint(pointToReach) ) // updates the next point, with obstacle avoidance, etc
    {
        signal(threadId,gmVariable(EVENT_FAILED)); // or be more specific if you want EVENT_PATHBLOCKED, EVENT_STUCK, etc
        return;
    }
    MicroThread::Yield();
}
signal(threadId,gmVariable(EVENT_SUCCESS)); // made it!


To accomodate the same thing all on the script side I don't think you need to do anything special. In that case the WalkToPoint function would just be a function that you call from script, and it would be structured the same way, but since you call it directly, and it can yield, sleep, etc. it won't return until finished either. In that sense it's just a normal function that happens to do asynchronous stuff.


Top
 Profile  
Reply with quote  
 Post subject: Re: Yielding From C Code
PostPosted: Wed Apr 14, 2010 3:36 pm 
Offline

Joined: Fri Apr 02, 2010 10:36 pm
Posts: 14
I went ahead and implemented it the way I discussed. GM creates a microthread for a cFunction if needed (if you don't register your function as requiring a microthread it uses exactly the old code path). If your microthread Yields() or Sleeps(), the GM threading code handles it and re-awakens the microthread the next time the GM thread runs. It works really well. Neither the script nor the native code need to do anything special. Either can call Yield() or Sleep(). It was a pretty small change, only really affecting gmThread::PushStackFrame.

I'll clean it up (I didn't use GM allocations and did no error checking), see about supporting Block(), test it more thoroughly and post it next week sometime. MicroThreadSleep is platform specific (it saves registers and swaps stack pointers). I'll include the PC version.

We'll definitely be using it in our current titles.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group