I'm just trying to think if there are any drawbacks to handling battle.net packets asynchronously. Here is what I mean:

[code]Packet 1 - Recieved
Buffer contents:
ff 0a 25 00 69 61 67 6f 00 52 41 54 53 20 30 20 ..%.iago.RATS 0
30 20 30 20 30 20 30 20 30 20 30 20 30 20 30 00 0 0 0 0 0 0 0 0.
69 61 67 6f 00 iago.
Length: 37
Packet 1 - Unknown packet recieved
Packet 2 - Recieved
Buffer contents:
ff 0f 3e 00 12 00 00 00 10 00 00 00 8c 00 00 00 ..>.............
00 00 00 00 0d f0 ad ba 0d f0 ad ba 69 61 67 6f ............iago
00 54 68 69 73 20 69 73 20 61 20 74 65 6d 70 6f .This is a tempo
72 61 72 79 20 6d 65 73 73 61 67 65 3a 00 rary message:.
Length: 62
Packet 3 - Recieved
Buffer contents:
ff 0f 87 00 12 00 00 00 10 00 00 00 8c 00 00 00 ................
00 00 00 00 0d f0 ad ba 0d f0 ad ba 69 61 67 6f ............iago
00 59 6f 75 72 20 63 6f 6e 6e 65 63 74 69 6f 6e .Your connection
20 61 64 64 72 65 73 73 20 69 73 20 32 30 37 2e address is 207.
31 36 31 2e 31 34 34 2e 31 36 32 3a 34 34 39 32 161.144.162:4492
38 2e 20 50 4c 45 41 53 45 20 49 4e 43 4c 55 44 8. PLEASE INCLUD
45 20 54 48 49 53 20 49 4e 20 41 4e 59 20 42 55 E THIS IN ANY BU
47 20 52 45 50 4f 52 54 2e 20 4d 61 6e 79 20 74 G REPORT. Many t
68 61 6e 6b 73 2e 00 hanks..
Length: 135
Packet 2 - Unknown packet recieved
Packet 3 - Unknown packet recieved
[/code]

For each packet that comes in, I create a process thread and let it go. Since I'm using Java, synchronization isn't difficult and overhead isn't really a big concern (since Java already has plenty of overhead) so I don't really care about those. Also, the length is used so each one is always handled entirely, I won't get parts shuffled.

Can anybody suggest any reason not to handle it like this, though? The only thing I can think of is that chat messages may not display in the correct order ever time; however, if two messages are sent closely enough that this occurs anyway, it's really not that big of a problem.

One thing I like it for is SID_PING, though, because it will instantly handle it, even while if it's processing SID_AUTH_INFO and SID_AUTH_CHECK.

Anybody else have any ideas for potential problems?

[quote author=iago link=board=17;threadid=5789;start=0#msg49526 date=1079373785]
I'm just trying to think if there are any drawbacks to handling battle.net packets asynchronously. Here is what I mean:

[code]Packet 1 - Recieved
Buffer contents:
ff 0a 25 00 69 61 67 6f 00 52 41 54 53 20 30 20 ..%.iago.RATS 0
30 20 30 20 30 20 30 20 30 20 30 20 30 20 30 00 0 0 0 0 0 0 0 0.
69 61 67 6f 00 iago.
Length: 37
Packet 1 - Unknown packet recieved
Packet 2 - Recieved
Buffer contents:
ff 0f 3e 00 12 00 00 00 10 00 00 00 8c 00 00 00 ..>.............
00 00 00 00 0d f0 ad ba 0d f0 ad ba 69 61 67 6f ............iago
00 54 68 69 73 20 69 73 20 61 20 74 65 6d 70 6f .This is a tempo
72 61 72 79 20 6d 65 73 73 61 67 65 3a 00 rary message:.
Length: 62
Packet 3 - Recieved
Buffer contents:
ff 0f 87 00 12 00 00 00 10 00 00 00 8c 00 00 00 ................
00 00 00 00 0d f0 ad ba 0d f0 ad ba 69 61 67 6f ............iago
00 59 6f 75 72 20 63 6f 6e 6e 65 63 74 69 6f 6e .Your connection
20 61 64 64 72 65 73 73 20 69 73 20 32 30 37 2e address is 207.
31 36 31 2e 31 34 34 2e 31 36 32 3a 34 34 39 32 161.144.162:4492
38 2e 20 50 4c 45 41 53 45 20 49 4e 43 4c 55 44 8. PLEASE INCLUD
45 20 54 48 49 53 20 49 4e 20 41 4e 59 20 42 55 E THIS IN ANY BU
47 20 52 45 50 4f 52 54 2e 20 4d 61 6e 79 20 74 G REPORT. Many t
68 61 6e 6b 73 2e 00 hanks..
Length: 135
Packet 2 - Unknown packet recieved
Packet 3 - Unknown packet recieved
[/code]

For each packet that comes in, I create a process thread and let it go. Since I'm using Java, synchronization isn't difficult and overhead isn't really a big concern (since Java already has plenty of overhead) so I don't really care about those. Also, the length is used so each one is always handled entirely, I won't get parts shuffled.

Can anybody suggest any reason not to handle it like this, though? The only thing I can think of is that chat messages may not display in the correct order ever time; however, if two messages are sent closely enough that this occurs anyway, it's really not that big of a problem.

One thing I like it for is SID_PING, though, because it will instantly handle it, even while if it's processing SID_AUTH_INFO and SID_AUTH_CHECK.

Anybody else have any ideas for potential problems?
[/quote]

I really like that idea... :o

You didn't have to quote that huge block, but thanks :)

I modelled it after how Starcraft handles game packets, except that I dispatch a new thread rather than wait for a processing thread to pick it up (or alerting it, anyway).

So, I guess it's not really modelled after Starcraft at all. Oh well :)

[quote author=iago link=board=17;threadid=5789;start=0#msg49534 date=1079375362]
You didn't have to quote that huge block, but thanks :)
[/quote]

I'v takin up the habit of quoting the authors post because in most cases it ends up getting deleted/modified way to much... :-\

[quote author=Eli_1 link=board=17;threadid=5789;start=0#msg49537 date=1079375562]
[quote author=iago link=board=17;threadid=5789;start=0#msg49534 date=1079375362]
You didn't have to quote that huge block, but thanks :)
[/quote]

I'v takin up the habit of quoting the authors post because in most cases it ends up getting deleted/modified way to much... :-\
[/quote]

That's fine, I would recommend replace the code tags with <CODE BLOCK> or something just to save me scrolling.

Anyway, back to the original point. :P

before we get back to the origional point...

[code]
'Using the CreateThread function in Visual Basic
'is very risky! VB5 is 'kinda' stable, but VB6
'applications will probably crash when you
'use the CreateThread function.
[/code]

those comments were taken from a CreateThread() example in VB6...
what's that all about?

[quote author=Eli_1 link=board=17;threadid=5789;start=0#msg49545 date=1079377119]
before we get back to the origional point...

<comments>

those comments were taken from a CreateThread() example in VB6...
what's that all about?
[/quote]

The problem with using threads is that the same variable can be modified by your program in more than one place. If your program isn't "thread safe" (ie, the variables aren't protected from this), it'll corrupt and destroy the variables.

Visual Basic probably isn't thread safe.

[quote author=iago link=board=17;threadid=5789;start=0#msg49534 date=1079375362]
You didn't have to quote that huge block, but thanks :)

I modelled it after how Starcraft handles game packets, except that I dispatch a new thread rather than wait for a processing thread to pick it up (or alerting it, anyway).

So, I guess it's not really modelled after Starcraft at all. Oh well :)
[/quote]

As a matter of fact, that is almost exactly how I process incoming packets. Here's how mine works (C# is almost identical to Java, so it's probably similar anyway):

1.) I create a thread-safe ArrayList
2.) I create a Timer that fires every 10 ms.
3.) Incoming packets are stuck into the thread-safe ArrayList as they come in. If an incoming packet is longer than the BNCS packet header says (the 2nd word), I split at the length of the packet before adding it to the ArrayList. If the BNCS packet header is invalid (doesn't begin with 0xff), it is ignored but still added to the ArrayList.
4.) The Timer fires its event, which causes the first packet to come out of the ArrayList (arrayList.RemoveAt(0)). If the packet is shorter than the BNCS headers say it should be, then the ArrayList is looked through for a packet without a valid header (which will occur in one of the first few). If it finds one, it combines them.

The big caveat here is the .NET event handler model -- it handles threading automatically, and I'm not sure how Java's threading model works. Each event gets its own thread spawned.

As you pointed out, the order of chat events isn't really crucial anyway, but I've never seen any extremely odd behavior with it.

Cheers.

that's actually very dissimilar to mine. When a packet is recieved, it creates a new instance of Processor, then I can Processor.start() which creates a new instance of the thread and returns immediately(ish). When the Processor is finished, the thread returns (ie, is over)/

[quote author=iago link=board=17;threadid=5789;start=0#msg49637 date=1079401244]
that's actually very dissimilar to mine. When a packet is recieved, it creates a new instance of Processor, then I can Processor.start() which creates a new instance of the thread and returns immediately(ish). When the Processor is finished, the thread returns (ie, is over)/
[/quote]

Your Processor class handles the packet parsing, correct? It is simply another path of execution for your program.... A funciton callback and a function called start() that serve the same purpose are relatively close :)

What I do, is I have a thread that is always checking for incomming data. Once it finds some, it saves it as a big byte array, terminated by a whole lot of nulls. Then I pass it through a function that shaves all of the null bytes off the end of the array and sends it to be parsed.
I dont bother dealing with parsing the packet length.
The only drawback from using this method is that a large byte array can take up a good deal of RAM. I doubt that it will be any problem for the computer that you are using.

[quote author=DaRk-FeAnOr link=board=17;threadid=5789;start=0#msg49687 date=1079417167]
What I do, is I have a thread that is always checking for incomming data. Once it finds some, it saves it as a big byte array, terminated by a whole lot of nulls. Then I pass it through a function that shaves all of the null bytes off the end of the array and sends it to be parsed.
I dont bother dealing with parsing the packet length.
The only drawback from using this method is that a large byte array can take up a good deal of RAM. I doubt that it will be any problem for the computer that you are using.
[/quote]

This sounds like it won't work if you happen to get a packet that is split, and where the last bytes of the first part are 00's.

[quote author=Adron link=board=17;threadid=5789;start=0#msg49781 date=1079457413]
[quote author=DaRk-FeAnOr link=board=17;threadid=5789;start=0#msg49687 date=1079417167]
What I do, is I have a thread that is always checking for incomming data. Once it finds some, it saves it as a big byte array, terminated by a whole lot of nulls. Then I pass it through a function that shaves all of the null bytes off the end of the array and sends it to be parsed.
I dont bother dealing with parsing the packet length.
The only drawback from using this method is that a large byte array can take up a good deal of RAM. I doubt that it will be any problem for the computer that you are using.
[/quote]

This sounds like it won't work if you happen to get a packet that is split, and where the last bytes of the first part are 00's.
[/quote]

Yes, especially when I first log on I frequently find that the channel lists are split or stacked.

Also, the way I deal with that is using blocking input in my recv thread that doesn't get through until enough bytes are recieved.

[quote author=Adron link=board=17;threadid=5789;start=0#msg49781 date=1079457413]
This sounds like it won't work if you happen to get a packet that is split, and where the last bytes of the first part are 00's.
[/quote]
You could always split the packets by every FF byte and send them to be parsed seperately. Additionally, a packet terminating null byte will never be significant, at least in any battle.net packet that I have seen.

[quote author=DaRk-FeAnOr link=board=17;threadid=5789;start=0#msg49845 date=1079467951]You could always split the packets by every FF byte and send them to be parsed seperately.[/quote]

IIRC, NullBot did this. As such, you could confuse it rather badly by sending a 0xff embedded in a talk event, causing it to believe it had encountered the next packet in the sequence.

[quote author=Kp link=board=17;threadid=5789;start=0#msg49862 date=1079470197]
[quote author=DaRk-FeAnOr link=board=17;threadid=5789;start=0#msg49845 date=1079467951]You could always split the packets by every FF byte and send them to be parsed seperately.[/quote]

IIRC, NullBot did this. As such, you could confuse it rather badly by sending a 0xff embedded in a talk event, causing it to believe it had encountered the next packet in the sequence.
[/quote]

What about somebody who has a ping of -1? ff 0f xx xx 00 00 00 00 00 00 00 00 ff ff ff ff would break it.

what about someone talking in D2 colors?

FF 63 31 77 6F 30 74 21 ÿc1wo0t!

Yes, both those things would confuse a bot which simply scanned for 0xff.

[quote author=DaRk-FeAnOr link=board=17;threadid=5789;start=0#msg49845 date=1079467951]
[quote author=Adron link=board=17;threadid=5789;start=0#msg49781 date=1079457413]
This sounds like it won't work if you happen to get a packet that is split, and where the last bytes of the first part are 00's.
[/quote]
You could always split the packets by every FF byte and send them to be parsed seperately. Additionally, a packet terminating null byte will never be significant, at least in any battle.net packet that I have seen.
[/quote]

A packet terminating null byte won't be significant, but one in the middle of a split packet will be.

Better yet, a D2 statstring is full of 0xff's if a character is naked (wearing no gear)

hmm, I think I might need to add some synchronization:

[quote]Login succeeded!
User in channel: pacinolife (Ping: 156, Flags: 0x0)
User in channel: 911kill (Ping: 63, Flags: 0x0)
User in channel: PaulMartin (Ping: 78, Flags: 0x0)
[....lots of users]
User in channel: starofblood04 (Ping: 62, Flags: 0x0)
User in channel: iago (Ping: 78, Flags: 0x10)
Info: Welcome to Battle.net!
Info: This server is hosted by AT&T.
Info: There are currently 67980 users playing 17174 games of Starcraft Broodwar,
and 196080 users playing 72459 games on Battle.net.
Info: Last logon: Wed Mar 17 4:28 PM
Joining channel: op [vL]
User in channel: [vL] (Ping: 63, Flags: 0x2)
User in channel: BinaryChat (Ping: 47, Flags: 0x0)
User in channel: '|'rance (Ping: 78, Flags: 0x0)
User in channel: tmp (Ping: 188, Flags: 0x0)
[....lots of users again.....]
User in channel: MacBinaryBot (Ping: -1, Flags: 0x10)
User in channel: iago (Ping: 78, Flags: 0x10)
Unknown packet recieved
Joining channel: Brood War CAN-1
User in channel: Naked_Zergling (Ping: 62, Flags: 0x0)
User in channel: ooooooooooo (Ping: 78, Flags: 0x0)
Ping recieved; returning
)[/quote]

Somehow, the channel joining packet ended up in totally the wrong place :)

lmao, I tried adding Java's version of a Critical Section around the 0x0F processing and it made some stuff worse and stuff stuff better:

[quote]User in channel: pacinolife (Ping: 156, Flags: 0x0)
User in channel: 911kill (Ping: 63, Flags: 0x0)
User in channel: Naked_Zergling (Ping: 62, Flags: 0x0)
Joining channel: Brood War CAN-1
User in channel: popIN (Ping: 47, Flags: 0x0)
User in channel: Em0kid (Ping: 31, Flags: 0x0)
User in channel: RockN_TiGeR (Ping: 32, Flags: 0x0)
User in channel: ooooooooooo (Ping: 78, Flags: 0x0)
User in channel: PaulMartin (Ping: 78, Flags: 0x0)
User in channel: Fr3EDoM (Ping: 46, Flags: 0x0)
Info: Last logon: Wed Mar 17 4:28 PM
Info: There are currently 65174 users playing 16390 games of Starcraft Broodwar, and 193485 users playing 71894 games on Battle.net.
User in channel: death_master90 (Ping: 31, Flags: 0x0)
User in channel: Halmir (Ping: 78, Flags: 0x0)
User in channel: thestingar (Ping: 266, Flags: 0x0)
User in channel: frankyc (Ping: 63, Flags: 0x0)
User in channel: iago (Ping: 63, Flags: 0x10)
Info: This server is hosted by AT&T.
User in channel: Knight_Lewis (Ping: 16, Flags: 0x0)
User in channel: DeerPoop99 (Ping: 78, Flags: 0x0)
User in channel: yiliangzhao (Ping: 484, Flags: 0x0)
User in channel: tiny_pest438 (Ping: 31, Flags: 0x0)
Info: Welcome to Battle.net!
User in channel: JO-BU (Ping: 47, Flags: 0x0)
User in channel: [vL] (Ping: 63, Flags: 0x2)
User in channel: '|'rance (Ping: 78, Flags: 0x0)
User in channel: tmp (Ping: 188, Flags: 0x0)
User in channel: [vL]Kp (Ping: 62, Flags: 0x0)
User in channel: [vL]Grok (Ping: 47, Flags: 0x0)
User in channel: thuscelackpiss (Ping: 109, Flags: 0x0)
User in channel: Zakath[vL] (Ping: 15, Flags: 0x0)
User in channel: Hostile[vL] (Ping: 0, Flags: 0x0)
User in channel: JoKeR[vL] (Ping: 0, Flags: 0x0)
User in channel: [vL]DarkVirus (Ping: 47, Flags: 0x0)
User in channel: hismajesty. (Ping: 32, Flags: 0x0)
User in channel: UserLoser. (Ping: 234, Flags: 0x0)
User in channel: naem@Azeroth (Ping: 93, Flags: 0x0)
User in channel: Eibro@Azeroth (Ping: 94, Flags: 0x0)
Joining channel: op [vL]
User in channel: BinaryChat (Ping: 47, Flags: 0x0)
User in channel: Yoni-L (Ping: 187, Flags: 0x0)
User in channel: Yoni-R (Ping: 453, Flags: 0x0)
User in channel: Skywing[vL]@Azeroth (Ping: 47, Flags: 0x0)
User in channel: Skywing[vL] (Ping: 0, Flags: 0x0)
User in channel: Telos.fe (Ping: 953, Flags: 0x0)
User in channel: Arta (Ping: 157, Flags: 0x0)
User in channel: Arta[vL] (Ping: 125, Flags: 0x0)
User in channel: cipher (Ping: 532, Flags: 0x0)
User in channel: Winner[vL] (Ping: 62, Flags: 0x0)
User in channel: iago (Ping: 63, Flags: 0x10)
User in channel: MacBinaryBot (Ping: -1, Flags: 0x10)
Ping recieved; returning
[/quote]

I think you need to set up some kind of event on on each incoming data packet to wait on before you process it (at least for chat events).

In fact, I really don't see why you are trying to do this asynchronously. Given the short processing time, it's likely that you will have more overhead associated with setting up the asynchronous requests than it will take to process the data.

[quote author=Skywing link=board=17;threadid=5789;start=15#msg50050 date=1079542264]
I think you need to set up some kind of event on on each incoming data packet to wait on before you process it (at least for chat events).

In fact, I really don't see why you are trying to do this asynchronously. Given the short processing time, it's likely that you will have more overhead associated with setting up the asynchronous requests than it will take to process the data.
[/quote]

That's also what I was thinking. It seemed like it could be interesting to set up, though. I might just go back to processing them the old fashioned way :)

ohwell, it was fun while it lasted. Fixed it now, though. It just does processing the old fashioned way.