I just finished coding the DTD for the XML, and also coded up two example XML-formatted packets. I'd love to hear your thoughts on it!

This XML example demonstrates basic usage:

[code]<?xml version="1.0"?>
<!DOCTYPE packet SYSTEM "http://labs.bnetdocs.org/dtd/packet.dtd">

<packet ordinal="0x15" name="SID_CHECKAD" direction="2" clients="star,sexp" relatedurls="type=bncs&amp;id=0x15&amp;direction=1">
<!-- relatedurls would be delimited by commas -->
<format>
<field type="DWORD" descr="Platform ID" />
<field type="DWORD"  descr="Product ID" />
<field type="DWORD"  descr="ID of last displayed banner" />
<field type="DWORD"  descr="Current Time" />
<!-- Following are just fake, meant as an example -->
<field type="STRING" descr="Null Terminated String" />
<field type="VOID" descr="Non-Null Terminated String" />
</format>
<remarks>
<!-- if any, can include html for formatting, but use [] instead of <> -->
Requests ad banner information from battle.net.
</remarks>
</packet>[/code]

This XML example demonstrates usage of loop, and conditionals:

[code]<?xml version="1.0"?>
<!DOCTYPE packet SYSTEM "http://labs.bnetdocs.org/dtd/packet.dtd">

<packet ordinal="0x09" name="SID_GETADVLISTEX" direction="1" clients="star,sexp" relatedurls="type=bncs&amp;id=0x19&amp;direction=2">
<!-- relatedurls would be delimited by commas -->
<format>
<field type="DWORD" descr="Number of Games" />
<field type="DWORD" case="if DWORD1 == 0" descr="Status" />
<repeat case="DWORD1 > 0" amtdefined="true" amt="1">
<!-- If amtdefined is set to true, tell it which field contains amount (Ex: amt="1" means first field) -->
<field type="WORD" descr="Game Type" />
<field type="WORD" descr="Parameter" />
<field type="DWORD" descr="Language ID" />
<field type="WORD" descr="Address Family" default="AF_INET" />
<field type="WORD" descr="Port" />
<field type="DWORD" descr="Host's IP Address" />
<field type="DWORD" amt="2" descr="Sin Zero (0)" />
<field type="DWORD" descr="Game Status" />
<field type="DWORD" descr="Elapsed time (in seconds)" />
<field type="STRING" descr="Game Name" />
<field type="STRING" descr="Game Password" />
<field type="STRING" descr="Game Statstring" />
</repeat>
</format>
<remarks>
<!-- if any, can include html for formatting, but use [] instead of <> -->
[p]Returns a list of available games and their information. Varies depending on product.[/p]
[p]Note that fields from Address Family to sin_zero form a sockaddr_in structure.[/p]
[u]Valid status codes:[/u]
[dl][dd]0x00: OK
0x01: Game doesn't exist
0x02: Incorrect password
0x03: Game full
0x04: Game already started
0x06: Too many server requests[/dd][/dl]
</remarks>
</packet>[/code]

This is the DTD both XML examples are validating against:

[code]<!ELEMENT packet (format, remarks)>
<!ATTLIST packet ordinal CDATA #REQUIRED>
<!ATTLIST packet name CDATA #REQUIRED>
<!ATTLIST packet direction CDATA #REQUIRED>
<!ATTLIST packet clients CDATA #REQUIRED>
<!ATTLIST packet relatedurls CDATA #REQUIRED>
<!ELEMENT format (field*, repeat*)>
<!ELEMENT field EMPTY>
<!ATTLIST field type CDATA #REQUIRED>
<!ATTLIST field descr CDATA #REQUIRED>
<!ATTLIST field amt CDATA #IMPLIED>
<!ATTLIST field case CDATA #IMPLIED>
<!ATTLIST field default CDATA #IMPLIED>
<!ELEMENT repeat (field*)>
<!ATTLIST repeat case CDATA #REQUIRED>
<!ATTLIST repeat amtdefined (true | false | TRUE | FALSE) #REQUIRED>
<!ATTLIST repeat amt CDATA #REQUIRED>
<!ELEMENT remarks (#PCDATA)>[/code]

I've been using http://validator.w3.org/check to validate my XML code against the DTD I coded, although I may modify it if you have better ideas on how the XML for the packets should be structured/laid out.

This thread is related to an old thread I posted over a year ago, which can be found here:

https://davnit.net/bnet/vL/index.php?topic=17219.0

Thoughts?

[quote author=Don Cullen link=topic=17636.msg179647#msg179647 date=1220346856]
Thoughts?
[/quote]

What does this have to do with battle.net bot development

What kind of project are you wanting to make using the documentation?

I think it's a damn good idea. This way, all the BNetDocs info will have a uniform display, making things easier to understand.

Move related URLs out of the main body to there own element.
packet ids as decimal [for easier importing]
default values?
response?
[code]<packet id="21" ordinal="0x15" name="SID_CHECKAD" direction="2">
        <client>STAR</client>
        <client>SEXP</client>
        <related type="bncs" id="0x15" direction="1" />
<format>
<field type="DWORD" descr="Platform ID" default="IX86" />
<field type="DWORD"  descr="Product ID" default="STAR" />
<field type="DWORD"  descr="ID of last displayed banner" />
<field type="DWORD"  descr="Current Time" />
<!-- Following are just fake, meant as an example -->
<field type="STRING" descr="Null Terminated String" />
<field type="VOID" descr="Non-Null Terminated String" />
</format>
<remarks>
<!-- if any, can include html for formatting, but use [] instead of <> -->
Requests ad banner information from battle.net.
</remarks>
        <response id="42" direction="1" />
</packet>[/code]

Building a bot where all you have to do to add an additional packetis add the XML sceam... that would be nice.

[quote author=Don Cullen link=topic=17636.msg179647#msg179647 date=1220346856]
[code]<!-- relatedurls would be delimited by commas -->
[/code][/quote]If you need to have a repeated field, I suggest you follow your later convention.  Move the related URLs out into a child tag, and have one instance of the tag per related URL.

[quote author=Don Cullen link=topic=17636.msg179647#msg179647 date=1220346856][code]
<!-- if any, can include html for formatting, but use [] instead of <> -->[/code][/quote]Although your method makes the remark a bit easier to read in a text editor, it would simplify display tools if the embedded HTML used quoted angle brackets.  For example, [code]&lt;b&gt;This will have an HTML b tag around it.&lt;/b&gt;[/code]

[quote author=Don Cullen link=topic=17636.msg179647#msg179647 date=1220346856]
[code]<format>
<field type="DWORD" descr="Number of Games" />
<field type="DWORD" case="if DWORD1 == 0" descr="Status" />
<repeat case="DWORD1 > 0" amtdefined="true" amt="1">
<!-- If amtdefined is set to true, tell it which field contains amount (Ex: amt="1" means first field) -->[/code][/quote]

This isn't consistent.  Also, a few minor improvements are needed if you want to be able to synthesize parsers from the XML description.  For Status, your case has the word if, but for the repeat directive, you don't.  Although a client could use different case handlers for field vs. repeat, it would be simpler if the two case attributes had the same internal structure.  Second, you refer to DWORD1, but never name it explicitly.  Anyone familiar with the packet format will know that you're referring to the DWORD labeled "Number of Games", but new readers might not figure it out -- and it's unnecessarily vague for any sort of parser generator.  I suggest adding an optional attribute name to field.  If present, the reader should save the value of name and use it for matching up future references.  Your block could then be rewritten as:[code] <field type="DWORD" name="dwGameCount" descr="Number of Games" />
<field type="DWORD" case="dwGameCount == 0" descr="Status" />
<repeat case="dwGameCount > 0">
[/code]
Looking back, your comment suggests you may have intended to solve this problem by having the repeat directive set some tracking variables.  I think my way is cleaner, though, since naming the fields makes it easier for people to match up the intended usage visually, as well as simplifying doing a search through large definitions to jump to all references to a conditional field.

@Hdx and Kp -- excellent feedback! Thanks!

@Hdx-- why decimal? The SQL database has an ID field for the ordinals (ie: 0x09), so it'd be easy to have PHP quickly locate the corresponding packet via type, ID, and direction... Although if you do think it'd make it easier for botmakers, I can always add a decimal attribute to the packet element?

@KP-- In regards to using &gt; and &lt;, I used [] because that's easier on people who write the documentation rather than typing out &gt;/&lt;, but I just realized I can just let them use <> when coding/writing the documentation, and have PHP change those to their corresponding equivalents so the XML will validate. Thanks for pointing it out. :-)

I've updated the DTD and the XML accordingly:

XML:
[code]<?xml version="1.0"?>
<!DOCTYPE packet SYSTEM "http://labs.bnetdocs.org/dtd/packet.dtd">

<packet type="bncs" id="0x09" name="SID_GETADVLISTEX" direction="1" >
   <client>STAR</client>
   <client>SEXP</client>
   <related type="bncs" id="0x09" direction="2" />
   <format>
      <field type="DWORD" name="dwGameCount" descr="Number of Games" />
      <field type="DWORD" case="dwGameCount == 0" descr="Status" />
      <repeat case="dwGameCount > 0">
         <field type="WORD" descr="Game Type" />
         <field type="WORD" descr="Parameter" />
         <field type="DWORD" descr="Language ID" />
         <field type="WORD" descr="Address Family" default="AF_INET" />
         <field type="WORD" descr="Port" />
         <field type="DWORD" descr="Host's IP Address" />
         <field type="DWORD" descr="Sin Zero (0)" />
<field type="DWORD" descr="Sin Zero (0)" />
         <field type="DWORD" descr="Game Status" />
         <field type="DWORD" descr="Elapsed time (in seconds)" />
         <field type="STRING" descr="Game Name" />
         <field type="STRING" descr="Game Password" />
         <field type="STRING" descr="Game Statstring" />
      </repeat>
   </format>
   <remarks>
      &lt;p&gt;Returns a list of available games and their information. Varies depending on product.&lt;/p&gt;
      &lt;p&gt;Note that fields from Address Family to sin_zero form a sockaddr_in structure.&lt;p&gt;
      &lt;u&gt;Valid status codes:&lt;/u&gt;
         &lt;dl&gt;&lt;dd&gt;0x00: OK
         0x01: Game doesn't exist
         0x02: Incorrect password
         0x03: Game full
         0x04: Game already started
         0x06: Too many server requests&lt;/dd&gt;&lt;/dl&gt;
   </remarks>
   <response type="bncs" id="0x09" direction="2" />
</packet>[/code]

This is the DTD the XML is validating against:

[code]<!ENTITY % HTML.Version "BNETDocs Packet Format v1.0">
<!ELEMENT packet (client*, related*, format, remarks, response*)>
<!ATTLIST packet type CDATA #REQUIRED>
<!ATTLIST packet id CDATA #REQUIRED>
<!ATTLIST packet name CDATA #REQUIRED>
<!ATTLIST packet direction CDATA #REQUIRED>
<!ELEMENT client (#PCDATA)>
<!ELEMENT related (#PCDATA)>
<!ATTLIST related type CDATA #REQUIRED>
<!ATTLIST related id CDATA #REQUIRED>
<!ATTLIST related direction CDATA #REQUIRED>
<!ELEMENT format (field*, repeat*)>
<!ELEMENT field EMPTY>
<!ATTLIST field type CDATA #REQUIRED>
<!ATTLIST field descr CDATA #REQUIRED>
<!ATTLIST field name CDATA #IMPLIED>
<!ATTLIST field case CDATA #IMPLIED>
<!ATTLIST field default CDATA #IMPLIED>
<!ELEMENT repeat (field*)>
<!ATTLIST repeat case CDATA #REQUIRED>
<!ELEMENT remarks (#PCDATA)>
<!ELEMENT response (#PCDATA)>
<!ATTLIST response type CDATA #REQUIRED>
<!ATTLIST response id CDATA #REQUIRED>
<!ATTLIST response direction CDATA #REQUIRED>[/code]

Any thoughts/feedback before I mark that up as completed and begin work on other parts of BNETDocs?

[quote]@Hdx-- why decimal? The SQL database has an ID field for the ordinals (ie: 0x09), so it'd be easy to have PHP quickly locate the corresponding packet via type, ID, and direction... Although if you do think it'd make it easier for botmakers, I can always add a decimal attribute to the packet element?[/quote]
I just figured something like: GetPacketStructure(xmlfile, DIR_INCOMING, SID_PING) would be simpler then GetPacketStructure(xmlFile, DIR_INCOMING, "0x" & Right("00" & Hex(SID_PING), 2)).
So ya, I figure it'd be easier for bot makers. Seine you brought it up my idea for this would to have the entire bot be run off the XML file. It'd be cool to see if it would be possible/elegant.
Just off the top of my head:
[code]ParsePacket(Buffer in){
  in.skip(1);
  id = in.removeByte();
  in.skip(2);
  XML struct = GetPacketStruct(incoming, bncs, id);
  Properties data = new Properties();
  for each e in struct.GetElements("\Format"){
    switch(e.get("type")){
      case "DWORD": i = in.removeDWord(); break;
      case "WORD": i = in.removeWord(); break;
      ...
    }
    data.addProp(struct.get("Name") & "." & e.get("Name"), i);
  }

  for each e in struct.GetElements("\Response"){
    Buffer out = new Buffer();
    XML outStruct = GetPacketStruct(outgoing, e.get("Type"), e.get("ID"));
    for each x in outStruct.getElements("\Format"){
      switch(x.get("Type")){
        case "DWORD": out.addDWord(props.get(x.get("Data")));
        ....
      }
    }
    switch(outStruct.get("type")){
      case "BNCS": bncs.send(out); break;
      case "BNLS": bnls.send(out); break;
    }
  }
}[/code]
EXA SID_PING:
[code]<packet type="BNCS" id="37" hex="0x25" direction="1" name="SID_PING">
  <format>
    <field type="DWORD" name="PingValue" descr="Ping Value" />
  </format>
  <response type="bncs" id="37" hex="0x25" direction="2" />
</blah>

<packet type="BNCS" id="37" hex="0x25" direction="2" name="SID_PING">
  <format>
    <field type="DWORD" name="PingValue" descr="Ping Value from incoming packet" value="SID_PING.PingValue" />
  </format>
</packet>[/code]

Forgive the complete and utter ugliness. But you get my point?
Of corse we would have to be a bit more verbose then that to handle things like value="SHA1(Config.Password)" but meh.

@Hdx-- Makes sense. I need the hex id there since that's what I use to look up the packet data in the SQL table. But I'll add the decimal attribute to the packet element. It'll look like this:

[code]<packet type="bncs" id="0x09" decid="9" name="SID_GETADVLISTEX" direction="1" >[/code]

Will that work for you?

[quote author=Don Cullen link=topic=17636.msg179864#msg179864 date=1221776200]
@Hdx-- Makes sense. I need the hex id there since that's what I use to look up the packet data in the SQL table. But I'll add the decimal attribute to the packet element. It'll look like this:

[code]<packet type="bncs" id="0x09" decid="9" name="SID_GETADVLISTEX" direction="1" >[/code]

Will that work for you?
[/quote]
Yup yup. ugh, you're going to make me make this bot arnt you -.-
Maybe when I get home.
wait, wouldnt a better idea be to encapsulate them all like so:
[code]<BNCS>
  <Send>
    <Packet id name hex>
    </Packet>
  </Send>
  <Recv>
    <Packet id name hex>
    </Packet>
  </Recv.
</BNCS>[/code]
Iono, don't think that would fit for your application. Humm...

Cool. As for making you make the bot; I'm not going to make you-- BUT you're welcome to do so, as it's certainly an interesting idea in principle. But of course, you don't have to, your time is to do with as you please. :-)

And besides, now I have the XML format ironed out, I have to work on the packet editor, then I have to reformat all of the packets. That'll take me eternity, considering how there's over 300 of them and none of them are in XML format. Fun, eh?

Edit: Just saw the edit you made; while I could do that, users could just request both the send/recv data and parse both if the user wanted both... This way, it'll save on bandwidth.

It's late.. What interesting bot idea am I completely oblivious to at the current moment in time?

EDIT: So basically an automated bot?

I suggested to Hdx that you have packetlog outputs so we can see what's going on as well (for learning).

[quote author=Sixen link=topic=17636.msg179879#msg179879 date=1221801475]
I suggested to Hdx that you have packetlog outputs so we can see what's going on as well (for learning).
[/quote]

That'd be kinda cool for the newer people to use to compare if they are having trouble.

I am interested in creating a bot using the methods suggested in this topic. My language of choice is Python, though if somebody can convince me to use something else, or if Hdx or whoever is starting their own implementation, I am flexible.

I'm actually working on writing a simple XML parser in C. I didn't find any that were cross platform what I could easily include in my project [i dont know why.. but i dont really want to add another dll]
If that annoys me to much, i'll move to java. So if you wana snag on either of those language give me a ring.