Incoming event message filter

[bkc56]

I wasn't sure where to put this since it's about client internals rather than a new feature suggestion. But if it needs to be moved to "suggestions", that's fine.

 

A common problem in battle is lag. This is especially true during invasions. Have 20+ players on screen and add 50+ critters, and the next thing you know you're in hades with no change to flee/restore/etc. The problem is more messages from the server than the client can keep up with. Command requests get stalled/ignored while it process all the messages.

 

It's my understanding (from talking with Grum) that a resync is the client dumping the backlog in an attempt to catch up. But those also "lock" up the client for several seconds with the same result: a quick trip to hades.

 

I was thinking: what if we filtered the messages before we get to the point of lag or resyncs? When you're in a battle you don't really care that much about stuff happening several squares away from you. In fact, you don't care that much about up-to-date graphics and movement around you as long as your stats (health, mana) are up-to-date and your commands (restore, use potions/magic/etc) are processed immediately.

 

So the question: as the incoming event queue begins to fill up, would it be possible to identify events that are happening farther away from you or are lower priority and ignore them? Keep the client from getting overwhelmed by throwing away stuff you don't care about that's farther away (or even off-screen if you're zoomed in).

 

You'd still need to get resynced eventually (perhaps when the queue is empty meaning the client is not behind anymore), but do it after the battle is finished and there's no harm in things freezing for a couple seconds.

 

So for those knowledgeable about the event handling and contents, would it be possible to do some sort of filtering? Provide some sort of trade-off between accurate screen contents and responsiveness to commands?

[alvieboy]

I wasn't sure where to put this since it's about client internals rather than a new feature suggestion. But if it needs to be moved to "suggestions", that's fine.

 

A common problem in battle is lag. This is especially true during invasions. Have 20+ players on screen and add 50+ critters, and the next thing you know you're in hades with no change to flee/restore/etc. The problem is more messages from the server than the client can keep up with. Command requests get stalled/ignored while it process all the messages.

 

It's my understanding (from talking with Grum) that a resync is the client dumping the backlog in an attempt to catch up. But those also "lock" up the client for several seconds with the same result: a quick trip to hades.

 

I was thinking: what if we filtered the messages before we get to the point of lag or resyncs? When you're in a battle you don't really care that much about stuff happening several squares away from you. In fact, you don't care that much about up-to-date graphics and movement around you as long as your stats (health, mana) are up-to-date and your commands (restore, use potions/magic/etc) are processed immediately.

 

So the question: as the incoming event queue begins to fill up, would it be possible to identify events that are happening farther away from you or are lower priority and ignore them? Keep the client from getting overwhelmed by throwing away stuff you don't care about that's farther away (or even off-screen if you're zoomed in).

 

You'd still need to get resynced eventually (perhaps when the queue is empty meaning the client is not behind anymore), but do it after the battle is finished and there's no harm in things freezing for a couple seconds.

 

So for those knowledgeable about the event handling and contents, would it be possible to do some sort of filtering? Provide some sort of trade-off between accurate screen contents and responsiveness to commands?

 

Hey

 

Well, there is no actual queue right now as far as I know, so when client gets a message from the server, it is dispatched immediately. That simplifies things because you don't need to worry about dependencies in the messages.

 

So what you're asking is to have a "delay queue" where all server messages that you think are not relevant at the moment are put in, and only processed after some time w/o a priority message.

 

This is very tricky and error-prone. And if (as I do believe *always* is) the network is slow, it won't help you at all. So, this would have to be done at server level - which is an almost impossible/time consuming/cpu consuming task.

 

Still, I might be wrong.

 

Al

[bkc56]

So what you're asking is to have a "delay queue" where all server messages that you think are not relevant at the moment are put in, and only processed after some time w/o a priority message.

No, I was actually suggesting that the less important message be thrown away (ignored) to keep the client responsive to user input/commands. A later resync can get everything corrected. I agree that trying to prioritize messages and save the less important ones for later would be very complicated and prone to errors (not worth the risk).

[crusadingknight]

Well, there is no actual queue right now as far as I know, so when client gets a message from the server, it is dispatched immediately. That simplifies things because you don't need to worry about dependencies in the messages.

Actually, there is such a queue - the actor->que member, which can hold up to MAX_CMD_QUEUE-2 actions before overflowing and forcing a resync. (See actor_scripts.c, line 1030 to see more on the command buffering.)

 

Another possibility might be to run through and apply all of the internal updates (mainly position) when the command queue starts getting past a given threshold, discard animations to catch up. The client can then request the resending of the actors, and update them on the fly (as the client already accounts for, to replace duplicates). This works better than dumping all of the actors, which causes the render lag while there is no 'self' actor to display (as used to happen during character creation when switching body parts, until I wrote that patch to swap body parts instead of destroying and recreating the actor.) Not a very hard fix (theoretically) either, and can probably be restricted to update_all_actors alone without touching any other code.

 

EDIT: @bkc56:

In fact, you don't care that much about up-to-date graphics and movement around you as long as your stats (health, mana) are up-to-date and your commands (restore, use potions/magic/etc) are processed immediately.

These actually aren't stored in the actor queue, but sent as seperate messages and processed immediately; Their lag only seems to be because of render lag caused by the lack of a self actor, or update lag from lack of a target actor.

 

EDIT 2:

In fact, now that I think about it, couldn't one technically update all of the internal state in a single iteration through the affected queue without applying any animations, and just queue the new commands without worrying about updating the full state from the server via update_all_actors()? This way, you technically keep the state consistent on both ends, without the network traffic or resyncs, with only the loss of an animation queue which was going to be purged anyway.

Edited October 19, 2007 by crusadingknight

[bkc56]

You'd probably want to still process the player's animations (so he can see if he's fighting, walking, whatever), but I could live without some/all of the other animations if it would stop the lag so I can restore.

 

I assume no animations would cause people to jump around rather than walking, and there's be no turning in-place or fighting animations going on.

 

But again, if you're computer is slow and can't keep up, missing animations is better than having the restore happen after you're gone to hades (because of command delays).

[crusadingknight]

You'd probably want to still process the player's animations (so he can see if he's fighting, walking, whatever), but I could live without some/all of the other animations if it would stop the lag so I can restore.

Well, if you were simply dumping from the affected queue as I described in the second (and optimal) solution, this would only happen when your own actor queue overflowed, at which point the some of animations _must_ be discarded anyway. The client will already speed up updates to catch up if it can, so getting to an overflow means there is nothing left to do. (Of course, I assume by lag you mean the freeze during resync.)

I assume no animations would cause people to jump around rather than walking, and there's be no turning in-place or fighting animations going on.

 

But again, if you're computer is slow and can't keep up, missing animations is better than having the restore happen after you're gone to hades (because of command delays).

Well, that's how the resyncs work now anyway, so it's either jump-and-freeze-while-actors-are-deleted-and-resent, or jump without freezing. Either way, the queues have to be partially purged to catch up once they start to overflow.

 

Postscript: Of course, I'm still uncertain about this solution, though I can't find anything wrong with the queue purging from looking at how it is currently handled - comments from someone on the server side would be appreciated.

Edited October 19, 2007 by crusadingknight

[Torg]

Postscript: Of course, I'm still uncertain about this solution, though I can't find anything wrong with the queue purging from looking at how it is currently handled - comments from someone on the server side would be appreciated.

If it is simply a graphics problem, which it sounds like given you are still processing the commands, just scraping the animations, then it's not really a server issue at all. That said, wider input is always a good thing. ;-)

[crusadingknight]

If it is simply a graphics problem, which it sounds like given you are still processing the commands, just scraping the animations, then it's not really a server issue at all. That said, wider input is always a good thing. ;-)

I'm just uncertain whether it could break anything by flushing the queue; For all I know the server could be using some heuristic on properly updating actors to detect illegal client. After all, if the queue can just be flushed and updated all at once, why isn't the client just doing that in the first place?

 

Not to mention, it's always the smallest things that are overlooked.

Edited October 20, 2007 by crusadingknight

[ttlanhil]

You get the same problem with 1v1 fights, if they last long enough (we spent a while testing longer fights with the crashes before the last update came out)... Large fights may make it more common, but it'll still happen.

The correct answer? Correct the timings.

If the server is sending actor commands faster than the client displays them (which it does, and for another point, see how soon you open a bag when you click on it from a distance, still a few steps away when it opens. That's the same thing, the client has the commands, but isn't processing fast enough), you'll always have problems from this, no matter how hard you try to engineer around them.

[bkc56]

Yet another #day/invasion with people having problems with lag. So...

 

Is there anything we can do with incoming messages to help the lag problem? Is anyone available to take ownership of this problem and makes some changes to the code?

 

(not being pushy, just don't want to see it fall through the cracks if there's a change to improve it)

[Torg]

Is there anything we can do with incoming messages to help the lag problem? Is anyone available to take ownership of this problem and makes some changes to the code?

I don't see the harm in either speeding up, or simply dropping excess animations gradually as the queue is getting larger, but I don't think we agree it will work yet, and we haven't had any input from Ent or Learner about the server handling of this.

 

It's also too late to get into this client update I expect. Therefore, I would suggest reviving the topic after the release and we can talk about it for the next release.

[crusadingknight]

I don't see the harm in either speeding up, or simply dropping excess animations gradually as the queue is getting larger, but I don't think we agree it will work yet, and we haven't had any input from Ent or Learner about the server handling of this.

 

It's also too late to get into this client update I expect. Therefore, I would suggest reviving the topic after the release and we can talk about it for the next release.

Actually, we already speed up the timer at line 932 of actor_scripts.c to cope. ttlhanil is correct that a good part of the fault seems to lie with the server timings, though it's still nice to be able to handle overflows caused by rare network problems or the like (bursts caused by packet loss, etc.).

[Learner]

I don't see the harm in either speeding up, or simply dropping excess animations gradually as the queue is getting larger, but I don't think we agree it will work yet, and we haven't had any input from Ent or Learner about the server handling of this.

 

It's also too late to get into this client update I expect. Therefore, I would suggest reviving the topic after the release and we can talk about it for the next release.

Actually, we already speed up the timer at line 932 of actor_scripts.c to cope. ttlhanil is correct that a good part of the fault seems to lie with the server timings, though it's still nice to be able to handle overflows caused by rare network problems or the like (bursts caused by packet loss, etc.).

Someone needs to check the animation times as they are used in the client. If animations are too long, then the trouble follows. The server has no concept of the length of animations, just can tick off the command for the next one at or after the fixed rate that things can happen at. I think anything over 1/2 second or 6/10ths of a second is too long. I haven't looked in that code in a long time.

[ttlanhil]

If every step taken (and even walking can show the mismatch between client and server in timings, like bags opening before you step on them) or action taken is the same length of time, server-side, then all it should take is the exact length of a tick for the server to normalise all client-side actor_command animations.

Theoretically, this shouldn't be hard to do or take long, but it can fix a range of glitches, so I'd suggest it gets fixed ASAP