If a packet is broken into segments, such as the 0x09 packet, what would be a good way of assosiating the broken segments into 0x09 when the first packet is the only one with the 0x09 header.

So basically, after I recieve the first segment (labeled 0x09), it's passed the the 0x09 handler - however, the segments after that aren't labeled.

I'm pretty sure I'm understanding you correctly. What you could do is create a buffer of some sort to store the values for a particular packet. So for example, if you have a packet that you are going to receive and you know that the packet is going to be say 20 bytes in length and the first byte is 2 in length, the second is also 2, the third is 4 and the other 12 are part of a single value. You could create an array of arrays (or another method if you so choose) to receive the packet until each array in this array is filled. Using this concept, you could have 1 array of arrays to represent each BNCS packet.

Using pseudo-code, say you have this array for 0x09:

[code]array id[2], length[2]
array some_value[4]
array packet_0x09[] = {id, length, some_value, another_value}[/code]

So you could copy data from the packet to the first array of packet_0x09[] until it is full (in this case, it can fit 2 bytes). Copy data to the second array til it's full, to the third, and to the fourth.

You could have some variable that holds the number of arrays within the packet_0x09 if you like. But either way, the purpose is to just fill each array within the main array as described above until each sub-array is filled. Note: In this case, another_value could be a char* and you could malloc() the proper amount of space prior to copying to it.

I thought of this idea a long time ago and had it all worked out, but when you asked this I just typed this right now as it came to me again. So you may have to change a few things, but this should work perfectly.

Just keep receiving data from the server and storing it into a buffer and add things to your packet arrays (as I described earlier) until they are filled.

Having packets split up is nothing new, nor is it 0x09 specific. You should rectify your code to wait on parsing that packet until all of the data is received. Check out my bot's handler for receive messages sent by WSAAsyncSelect():


[code]
globally declared:

int recvlen;
char wsbuffer[4096];
char *wsdata = wsbuffer;

....


if (LOWORD(lParam) == FD_CLOSE) {
AddChat(vbRed, "[BNET] Winsock Close.", hWnd_rtfChat);
int err = HIWORD(lParam);
if (err) {
sprintf(asdf, "[BNET] Winsock Error %d !", bnetlen);
AddChat(vbRed, asdf, hWnd_rtfChat);
sprintf(asdf, "Winsock error %d encountered!", bnetlen);
PopupTrayMsg("Connection dropped!", asdf, NIIF_ERROR);
}
Disconnect(false); 
return 0;
}
int tmp = recv(wParam, wsdata + recvlen, sizeof(wsbuffer) - (wsdata - wsbuffer) - recvlen, 0);
if (tmp == SOCKET_ERROR) {
sprintf(asdf, "recv() error %d!", WSAGetLastError());
AddChat(vbRed, asdf, hWnd_rtfChat);
return 0;
}
recvlen += tmp;
while (recvlen >= 4) {
if ((unsigned char)*wsdata != (unsigned char)0xFF) {
sprintf(asdf, "[BNET] Parse error! expected 0xFF, 0x%02x.", (unsigned char)*wsdata);
AddChat(vbRed, asdf, hWnd_rtfChat);
return 0;
}
bnetlen = *(unsigned short *)(wsdata + 2);
if (bnetlen > recvlen) {
sprintf(asdf, "[BNET] Expecting %d more bytes.", bnetlen - recvlen);
AddChat(vbYellow, asdf, hWnd_rtfChat);
if (wsdata + recvlen >= wsbuffer + sizeof(wsbuffer)) {
memcpy(wsbuffer, wsdata, recvlen);
wsdata = wsbuffer;
}
return 0;
}
ParsePacket(wsdata[1], wsdata);
recvcount++;
recvlen -= bnetlen;
wsdata += bnetlen;
}
wsdata = wsbuffer;
recvlen = 0;
return 0;
[/code]

They posted pretty much the general idea you have to follow. But my question is this, are you referring to the BNCS or W2GS 0x09? If you are referring to the W3GS one does it not have the packet length somewhere inside of the data if it is variable? [post the sepcs you're working with if its not BNCS 0x09 please]

Assuming he meant BNCS 9, the game listing message, every one of the responders seems to have forgotten that this message has the property that it contains many records within it.  It has a fixed length header, followed by a variable number of variable length records, one per game enumerated.  The same basic principles used for reassembling PDUs from a TCP stream apply here, though of course the rules for what constitutes a record differ a bit from what constitutes a PDU.

Yes, but if it was BNCS 0x09 everyone above has it correct, sorta. The BNCS header for 0x09 works jsut the same as everyone else's, and includes the length of ALL the data. I *think* {guessing from the question and a few of his earlier posts} that he is referring to W3GS 0x09 which IIRC is the one used to transfer map data. Of which I am not 100% sure of the format, so I asked for it.

[quote author=Hdx link=topic=17763.msg180987#msg180987 date=1230705768]
Yes, but if it was BNCS 0x09 everyone above has it correct, sorta. The BNCS header for 0x09 works jsut the same as everyone else's, and includes the length of ALL the data.
[/quote]

Yes, but after he has the whole BNCS 9 message reassembled via the header length, he needs to go through [u]another[/u] round of parsing to pull out the individual game records, and there is no separate length field for those.  There is a count field, which is sufficient when combined with a buffer that knows the total PDU size.  brew's instructions are completely inappropriate for the subextraction, and Yegg's code doesn't look too relevant either.

Looking back at the original message, I think the poster needs to answer your question, tell us what language he's using, and give us some sample data.  He's too vague.

[quote author=Kp link=topic=17763.msg180990#msg180990 date=1230758234]
Yes, but after he has the whole BNCS 9 message reassembled via the header length, he needs to go through [u]another[/u] round of parsing to pull out the individual game records, and there is no separate length field for those.  There is a count field, which is sufficient when combined with a buffer that knows the total PDU size.
[/quote]

I'm not quite following you. You do realize (in my code) it's not being parsed until the entirety of the packet is there in the buffer, right? The size of each individual entry would therefore be irrelevant.
I took the OP's post to mean reassembling the primary packet, nothing to do with subextraction at all.

[quote author=Kp link=topic=17763.msg180990#msg180990 date=1230758234]
[quote author=Hdx link=topic=17763.msg180987#msg180987 date=1230705768]
Yes, but if it was BNCS 0x09 everyone above has it correct, sorta. The BNCS header for 0x09 works jsut the same as everyone else's, and includes the length of ALL the data.
[/quote]

Yes, but after he has the whole BNCS 9 message reassembled via the header length, he needs to go through [u]another[/u] round of parsing to pull out the individual game records, and there is no separate length field for those.  There is a count field, which is sufficient when combined with a buffer that knows the total PDU size.  brew's instructions are completely inappropriate for the subextraction, and Yegg's code doesn't look too relevant either.

Looking back at the original message, I think the poster needs to answer your question, tell us what language he's using, and give us some sample data.  He's too vague.
[/quote]
What do you mean subextraction? There is nothing different in 0x09 as any other packet. I think you're confusing me, From my understanding you're saying you have to extract more data from the socket then specified in the BNCS header, which, you don't.
As for extracting the data from each of the games listed, its a simple for loop (for x = 0; x < data.gamecount; x++) {remove shit} not hard

[quote author=brew link=topic=17763.msg180991#msg180991 date=1230764946]
I'm not quite following you. You do realize (in my code) it's not being parsed until the entirety of the packet is there in the buffer, right? The size of each individual entry would therefore be irrelevant.
I took the OP's post to mean reassembling the primary packet, nothing to do with subextraction at all.
[/quote]

I took it to mean that he couldn't figure out how to access over the individual records within the packet.  We approached different problems.

[quote author=Hdx link=topic=17763.msg180992#msg180992 date=1230766018]
What do you mean subextraction? There is nothing different in 0x09 as any other packet. I think you're confusing me, From my understanding you're saying you have to extract more data from the socket then specified in the BNCS header, which, you don't.[/quote]
I meant no such thing.

[quote author=Hdx link=topic=17763.msg180992#msg180992 date=1230766018]
As for extracting the data from each of the games listed, its a simple for loop (for x = 0; x < data.gamecount; x++) {remove shit} not hard
[/quote]

Removing individual records isn't entirely trivial since they don't have an internal length count, so you have to step over the contained strings to find the boundary, but no, it's not hard.  On the other hand, neither is reassembling the main PDU, but look how many people can't figure that out. :)

[quote author=Kp link=topic=17763.msg180994#msg180994 date=1230787147]Removing individual records isn't entirely trivial since they don't have an internal length count, so you have to step over the contained strings to find the boundary, but no, it's not hard.  On the other hand, neither is reassembling the main PDU, but look how many people can't figure that out. :)[/quote]Yar, do me a favor cuz i cant think right now, define PDU.
Anyways, I think I understand you correctly, due to it containing NT Strings. {AE: variable length data} it is difficult without the proper habbits {buffer classes} to obtain the data. On the other hand if it was simply x number of fixed size objects it'd be simple. {AE if each item had nothing but 5 dwords you'd know that you needed count * 20 bytes of data}
BUT, it still stands, that to obtain the entire packet load you would use the length in the BNCS header neh?

[quote author=Hdx link=topic=17763.msg180995#msg180995 date=1230792809]
Yar, do me a favor cuz i cant think right now, define PDU.
Anyways, I think I understand you correctly, due to it containing NT Strings. {AE: variable length data} it is difficult without the proper habbits {buffer classes} to obtain the data. On the other hand if it was simply x number of fixed size objects it'd be simple. {AE if each item had nothing but 5 dwords you'd know that you needed count * 20 bytes of data}
BUT, it still stands, that to obtain the entire packet load you would use the length in the BNCS header neh?
[/quote]

Protocol Data Unit.  Many posters here refer to it simply as a "packet," but I try to avoid that since it can confuse people since BNCS PDUs don't have any fixed mapping to TCP packets.  It comes up a fair bit in Wireshark documentation, though looking at the Wikipedia page for PDU, it seems my use for application data isn't very common, and so could be misleading in its own right.

Yes, you still need the BNCS header to know when you have the entire PDU.  When I first entered the thread, I assumed he'd gotten past basic PDU reassembly if he was to the point of looking at the game query packet.  I could be wrong on that, though.

Well it seems that it's no longer broken into segments.

I have been receiving all the data in one packet starting 2 days ago. I don't know why but I rather not ask questions and just let it be.

Late but any how, during my research at the time it is true at times the server will send you data in broken clumps, as if the server is queuing out the send data, which i would assume it does for reasons that dose not really matter on our end.

It all comes down to how you are handling the packets in the first place, your packet receiver should be checking if the packet is complete before processing through it, note that any packet that comes in broken the data to complete that packet will follow up in the servers next sends.
Fixing your packet processor to check packet length to see if you have a complete packet before processing the data will prevent you from having problems down the road.