I've been curious recently whether it would be feasable to make a C# bot download patch files and extract the version check files and replace the old ones using MBNCSutil's bnftp and mpq classes. I've already got it to successfully download the MPQ file using the name provided in the version check fail message, but I don't know how to find the hash files in there. (the only known files are exe and the (attributes), (listfile), and (signature) (according to the MPQ utility I'm using).

This is with the 1.15.3 patch for StarCraft: Broodwar.

UL knows all about it. It's a well kept secret, apparently.

[quote author=Andy link=topic=17663.msg179871#msg179871 date=1221785385]UL knows all about it. It's a well kept secret, apparently.[/quote]I saw that topic, but yeah, all I know is its an MPQ and it has no files in the listfile so I can't tell which files are the hashes.

Actually, I looked at the WarCraft III patch (last one that occurred) and saw that it has a file called Patch_WAR3.mpq inside the mpq, and that has the patch files inside it... but for some reason StarCraft's patch doesn't have the names. :/

They're different systems.

I guess that is logical with the observed results, however that doesn't help much in learning about patching StarCraft with the MPQ.

I don't want to go and make an ugly big function that calls StarCraft to do it, like people did with Warden before solving* that. Besides, this isn't as hard as Warden, is it? :O Maybe it is, but I've seen people have already figured out how MPQs store stuff.

You could just cheat and patch the game via you're bot by taking the mpq file, for example:
SEXP_IX86_1152_1153.mpq
Replace ".mpq" with ".exe"
Take "1153.exe", somthing like Tag = StrReverse(GetSTRING(StrReverse(File), "_"))
Connect to "ftp.blizzard.com"
Download "/pub/broodwar/patches/PC/BW-" & Tag
Execute the exe.
Wait awhile or track the patching progress.
Copy the binarys from you're starcraft folder to the bot's folder (if thats how you're bot works)
Take a guess at the version byte, based on the old version byte.
blah blah blah.

I had a little look into extracting the binarys once, but iirc, the files dont have a PE header so i'm not sure whats up with that -- might of been an error extracting them, it was along time ago.
But yeah, if I really wanted to add auto patching, I would just do the above, it's simple, it work's and to some degree, its fail safe(?)
The patch exe's have the same MPQ you would download from battle.net in them anyway -- it just appends the executable.
If you REALLY want to do it the hard way, I advise reverseing the patch exe to see what it does -- it should all be in there.

Uh... except that requires the actual game anyway...

[quote author=Ringo link=topic=17663.msg179890#msg179890 date=1221879931]You could just cheat and patch the game via you're bot by taking the mpq file, for example:
SEXP_IX86_1152_1153.mpq
Replace ".mpq" with ".exe"
Take "1153.exe", somthing like Tag = StrReverse(GetSTRING(StrReverse(File), "_"))
Connect to "ftp.blizzard.com"
Download "/pub/broodwar/patches/PC/BW-" & Tag
Execute the exe.
Wait awhile or track the patching progress.
Copy the binarys from you're starcraft folder to the bot's folder (if thats how you're bot works)
Take a guess at the version byte, based on the old version byte.
blah blah blah.

I had a little look into extracting the binarys once, but iirc, the files dont have a PE header so i'm not sure whats up with that -- might of been an error extracting them, it was along time ago.
But yeah, if I really wanted to add auto patching, I would just do the above, it's simple, it work's and to some degree, its fail safe(?)
The patch exe's have the same MPQ you would download from battle.net in them anyway -- it just appends the executable.
If you REALLY want to do it the hard way, I advise reverseing the patch exe to see what it does -- it should all be in there.[/quote]That's the kind of work-around I wanted to avoid because I have a hunch the public patches don't work if StarCraft isn't installed.

[quote author=Ringo link=topic=17663.msg179890#msg179890 date=1221879931]I had a little look into extracting the binarys once, but iirc, the files dont have a PE header so i'm not sure whats up with that -- might of been an error extracting them, it was along time ago.
But yeah, if I really wanted to add auto patching, I would just do the above, it's simple, it work's and to some degree, its fail safe(?)
The patch exe's have the same MPQ you would download from battle.net in them anyway -- it just appends the executable.
If you REALLY want to do it the hard way, I advise reverseing the patch exe to see what it does -- it should all be in there.[/quote]Might I suggest the idea that if it's actually a patch file, there wouldn't be any complete binary file in it. If there were binary files in there ready to simply extract, it would simply be a compressed and/or encrypted storage file.

I suggest you look into actually reading the data in the MPQ and actually patching your binaries. I'm unfamiliar with whatever patching mechanism Blizzard uses. Some companies use RTPatch (JFGI) rather than making their own proprietary patching system, so that could be a good starting point for comparison of reverse-engineered code.

If you're gonna reverse engineer anything, wouldn't BNUpdate.exe be the file to look at?

[quote author=Ribose link=topic=17663.msg179892#msg179892 date=1221883509]That's the kind of work-around I wanted to avoid because I have a hunch the public patches don't work if StarCraft isn't installed.
[/quote]
True, it was just an option :P
I haven't really looked into this in detail, but i'm sure you only need a few starcraft files in a directory and maybe a registery value or 2 -- if stardat etc needs to be there, then yeah, to much hastle, I guess.

[quote author=tA-Kane link=topic=17663.msg179893#msg179893 date=1221884061]
Might I suggest the idea that if it's actually a patch file, there wouldn't be any complete binary file in it. If there were binary files in there ready to simply extract, it would simply be a compressed and/or encrypted storage file.

I suggest you look into actually reading the data in the MPQ and actually patching your binaries. I'm unfamiliar with whatever patching mechanism Blizzard uses. Some companies use RTPatch (JFGI) rather than making their own proprietary patching system, so that could be a good starting point for comparison of reverse-engineered code.

[/quote]
Well, it was some years ago I tryed this, I  wunderd if the binarys would be extracted and ready to run, but after extracting them from the patch mpq, they didn't look like they had a PE header.
At that point, I didn't look into it in any more depth -- it was mainly just to answer a question.


[quote author=Andy link=topic=17663.msg179894#msg179894 date=1221884396]
If you're gonna reverse engineer anything, wouldn't BNUpdate.exe be the file to look at?
[/quote]
Yes, but the patch exe would be the better place to start I would have thought, since BNUpdate would require not only the game to be installed, but also running.
I haven't looked at the patch exe, but i'm pretty sure it creates a fake starcraft window, extracts the BNupdate.exe and runs it.
I'm sure theres alittle/alot more to it than that, but I persionaly would start with the patch exe it's self before poking around with BNUpdate.exe.
I think it would make more sence to try getting away with executing the BNUpdate.exe and let it do all the work, since BNUpdate comes with the patch, you never know for sure if it's going to follow a standard.

I might poke around with this if I get really bored some day starting with the patch exe -- I figger, if you know everything you need to run BNUpdate.exe then it's just a case of supplying it and running the update exe.
I think even doing it that way tho, blizzard could get bitchy and change BNUpdate to check for more install related stuff, making it harder to run.
If that was the case, you will run into bigger problems when trying to patch the binarys with you're own code, since I think BNUpdate only knows how and what to do for a said patch.
I really don't know a great deal about this, so don't quote me on this stuff.

No, I mean... BNUpdate tends to be one of the only executable files in the patch MPQ, so I'm fairly certain it IS the patch exe.

[quote author=Andy link=topic=17663.msg179896#msg179896 date=1221886588]
No, I mean... BNUpdate tends to be one of the only executable files in the patch MPQ, so I'm fairly certain it IS the patch exe.
[/quote]
It is, but, for example, BW-1153.exe contains the patch mpq and set's everything up, so BNUpdate.exe will run.
I just tested a few things, and the only things needed to patch brood war, are starcraft.exe, stardat.mpq and broodat.mpq.
BNUpdate.exe wont run, unless starcraft is running. So I figger that is what the patch exe does, creates a fake window.
From what I can see, the main reassion you need the stardat.mpq and broodat.mpq, is somthing to do with StarEdit.exe.
Every other file is created when patching (or extracted from the dat mpq's, then patched)
You could possibly get away with building smaller dat mpqs with just the needed files in them for bnupdate.exe to run, but i'm not sure how easy that would be.
Just by doing these few tests, it looks like the patch exe checks if starcraft's install folder exists, extracts the bnupdate.exe to it, creates a fake starcraft window, then executes the BNUpdate.exe.
Then what it looks like BNUpdate.exe is doing, is checks "starcraft.exe", "stardat.mpq" and "broodat.mpq" exists, before patching/doing it's thing.
If you can emulate the patch exe, in theory, you could just extract and run the bnupdate.exe -- that's still going to require you have the dat mpq's tho.
Emulating the whole BNUpdate.exe, in my eyes, could possibly be more hastle than it's worth, but at least, you wouldnt need 100mb of dat mpq's, since you would just be emulating the needed functions to patch the 3 checkrevision binarys.
Running BNUpdate.exe seems the better option for somthing like a BNLS server, where as emulating the BNUpdate.exe functions, seems a better option for a battle.net bot.
I still think, emulating bnupdate.exe is going to cause many problems, each time their is a patch -- that kind of defeats the object of automating patching, if it need's to be updated each patch, to patch :P
I'm guessing that's somthing skying and yoni's BNLS suffer's from, today?

That's what I would do anyway, if I was releaseing a battle.net bot, able to self patch it's binary's.
I would emulate pretty much what the patch exe does, and simply say, you must have the required dat mpq's inorder for it to patch, other wise it will fail.

I guess you have the 3 options:
A) Download the patch exe from ftp.blizzard.com and run it. (requires game to be installed, but most fail safe method)
B) Emulate the patch exe functions, and extract and run BNUpdate.exe from the mpq you download from battle.net. (requires the dat mpqs)
C) Emulate the bnupdate.exe enough to patch the 3 checkrevision binarys. (may require reworking, each patch or 3)

I persionaly would go for option B, but option A seems immune to any bnupdate changes, option C seems the most likely to suffer from changes to bnupdate, where as option B seems somwhere in the middle.
That's just me tho.

Bnupdate.exe is what you gotta look at.  It took me only two days to reverse and figure out to get it running correctly about two years ago, as I'm still the only person I know of who has running working code for this besides Skywing.  mpqs.lst, revert.lst, patch.cmd are some useful files that my patcher extract.  mpqs.lst will tell you if there are any MPQ files that have the version check files in them (only useful on War3 patches usually).  There's several types of patch types, 0, 1, 2, 4.  0 is a bit complex, 1 is the simplest as it is a direct copy, 2 and 4 (WoW/War3) are newer binary sort differential stuff.

patch.cmd is useful because it has things like "This patch upgrades Diablo II from version 1.00 or later to version 1.10." for example.

Also you need revert files if you plan on using update-from-any patches (PROD_PLAT_1xx_VERSION.mpq) instead of regular updates.  Revert files are the very first game version files (i.e. v 1.00).  On the update-from-any patches you actually have to copy over your existing game files with the revert files then patch those (that's how it allows you to update from any version)

From 5 mins of looking at it, 1 is extramly simple. [as you said]
(word) header len 0x18
(byte) unknown 0x04
(byte) ver
For 0, theres 8 bytes always 0, [checked 4 diffrent patches]
then u16 length of file data
and after that I have no clue what the data is, but i presume it's a crc or something knowing that bnupdate complains if the files dont match the old one, even on ver 1.
Havednt looked at 0 yet, but in the last patch only battle.snp was 0. So meh.

[quote author=Ribose link=topic=17663.msg179870#msg179870 date=1221784669]
I've been curious recently whether it would be feasable to make a C# bot download patch files and extract the version check files and replace the old ones using MBNCSutil's bnftp and mpq classes. I've already got it to successfully download the MPQ file using the name provided in the version check fail message, but I don't know how to find the hash files in there. (the only known files are exe and the (attributes), (listfile), and (signature) (according to the MPQ utility I'm using).

This is with the 1.15.3 patch for StarCraft: Broodwar.
[/quote]

Sorry I just saw this topic.

I've been toying with the diffpatch algorithm recently in an attempt to not have it for my upcoming bot but also to fix/update BNLS.  If my results are adequate they'll be rolled into MBNCSUtil.

[quote author=Hdx link=topic=17663.msg179927#msg179927 date=1222202995]
From 5 mins of looking at it, 1 is extramly simple. [as you said]
(word) header len 0x18
(byte) unknown 0x04
(byte) ver
For 0, theres 8 bytes always 0, [checked 4 diffrent patches]
then u16 length of file data
and after that I have no clue what the data is, but i presume it's a crc or something knowing that bnupdate complains if the files dont match the old one, even on ver 1.
Havednt looked at 0 yet, but in the last patch only battle.snp was 0. So meh.
[/quote]
I'm dubious about a u16 for file length - that means the length is capped at 64KiB?

ive encountered a few 1 files that have unknown value in the 2 bytes following the length.
But I've never encountered a file using 1 that was larger then 0xf000
Which would make sense as they would probaly use 0, the dif function that would significantly decrease the size of the file in the archive. But this is pure speculation so dont mind me head. I still havent actually looked at bnupdate

[code]
typedef struct patchfileheader_t {
WORD SizeOfStructure; // Should always be 0x18 (24)
BYTE Version; // ? 4: Always seen to be 4
BYTE PatchType; // 0: Differential, 1: Direct Copy, 2/4: Binary sort differential
DWORD Checksum; // CRC32 checksum of filedata with length of GetFileSize() on old file
DWORD OldFileSize; // What old file size should be (if this is 0, ignore)
DWORD NewFileSize; // What new file size should be (if this is 0, ignore)
FILETIME NewFileTime; // What to set new FILETIME to
} PATCHFILEHEADER;
[/code]

I'm a bit confused on the Checksum... It's a check on the old file, on what the new file should be, or the data after the header? And what does "length of GetFileSize() on old file" mean?

[quote author=Andy link=topic=17663.msg179974#msg179974 date=1222526062]
I'm a bit confused on the Checksum... It's a check on the old file, on what the new file should be, or the data after the header? And what does "length of GetFileSize() on old file" mean?
[/quote]

Well, now that you have the patch file's header, it should be fairly easy to look at a patch file, and determine what exactly those mean.

humm, filetime you say... Wow i'm stupid :P never thought to check that.
Dosent GetFileSize() return the same value as Right-click -> properties ->File Size
And GetCompressedFileSize() return Size on Disc?
If so... I must of gotten my reverted battle.snp file wrong. [extracted from stardat.mpq]
As mine is 239,870 bytes CRC32 0x5207A451, and according to the patch file it should be 282878 CRC32 0x6B8F7726
[code]18 00 04 00 26 77 8F 6B FE 50 04 00 FE 80 08 00
A9 69 4B 0D AF EE C8 01[/code]neh?

[quote author=Andy link=topic=17663.msg179894#msg179894 date=1221884396]
If you're gonna reverse engineer anything, wouldn't BNUpdate.exe be the file to look at?
[/quote]

Should look at storm's ordinal 120 as well

[quote author=brew link=topic=17663.msg180032#msg180032 date=1222986191]
[quote author=Andy link=topic=17663.msg179894#msg179894 date=1221884396]
If you're gonna reverse engineer anything, wouldn't BNUpdate.exe be the file to look at?
[/quote]

Should look at storm's ordinal 120 as well
[/quote]

why

[quote author=UserLoser link=topic=17663.msg180090#msg180090 date=1224172670]
[quote author=brew link=topic=17663.msg180032#msg180032 date=1222986191]
[quote author=Andy link=topic=17663.msg179894#msg179894 date=1221884396]
If you're gonna reverse engineer anything, wouldn't BNUpdate.exe be the file to look at?
[/quote]

Should look at storm's ordinal 120 as well
[/quote]

why
[/quote]

Because it's what actually extracts the patch, creates the bnupdate process, etc?
It doesn't hurt to look.

dunno son, i never looked at that in my life

Sry, my english was not good enough to read 2 pages :)

Do you want to download patch from BNET and get a fresh exe's? What do you need it for? AFAIK local hashing doesn't work anymore.

Local hashing works just fine.

For BNET?

Can you give me a bot that works without extra connections?

Er, what? I have been able to easily make a bot that does not use BNLS (your "remote connections"), with the help of MBNCSutil, which is just a dll.
I do agree that there aren't many public bots that connect on their own, due to recent patches (StealthBot, for example).  But adversely, I know many that do (Andy's RCB does IIRC, and ringo made that SCGP thing that does not use BNLS, or remote connections for Warden support either, for that matter).

Ringo's bot use BNLS.

And probably later blizzard change something and your method dont work, like it was with stealth bot.

And can you give me a working example that uses BNET connection and dont use any other?

[quote author=xpeh link=topic=17663.msg181186#msg181186 date=1231973571]
Ringo's bot use BNLS.

And probably later blizzard change something and your method dont work, like it was with stealth bot.

And can you give me a working example that uses BNET connection and dont use any other?
[/quote]

Local hashing works just fine.

Where?

My bot. Local hashing works 100% fine. I'm actually working on a stripped down version right now, without all the excessive features and commands. Chat-only.

[quote author=xpeh link=topic=17663.msg181186#msg181186 date=1231973571]
Ringo's bot use BNLS.

And probably later blizzard change something and your method dont work, like it was with stealth bot.

And can you give me a working example that uses BNET connection and dont use any other?
[/quote]
Does it?
Well, as I thought, Andy's works fine, and as I know, mine works fine (as I tear the thing apart... :/)

Blizzard patches their clients, and when they do, one just updates the game files from the game's install directory (after applying the patch), since the patch system is not easy/practical to emulate (however, people have done it..., and I believe its easier to emulate WarCraft III's system than the other clients').

If they change something, like adding lockdown, then people just have to converse here, solve it, and add it to their bots (MBNCSutil, again, has support for it!).

Haha, StealthBot hasn't had a release in 3 years. That's a lot of time in terms of the Internet. I don't recall if they even had ver-IX86-#.mpq support (still on IX86ver#.mpq), let alone lockdown-IX86-##.mpq support in the public release. :/
With well supported bots and/or well made bots, one doesn't have to wait so long for such a patch.

I dont think getting patches from bnet and ripping exes from them is a good idea. It's much easier to download it somewhere from internet.

[quote author=xpeh link=topic=17663.msg181209#msg181209 date=1231992514]
I dont think getting patches from bnet and ripping exes from them is a good idea. It's much easier to download it somewhere from internet.
[/quote]
The game clients download the patches inside MPQ files. If one knew exactly what the client does, they can emulate that just like they do the rest of the Battle.net protocol (Blizzard didn't just give people the source to the clients, people reversed engineered the protocols). In the context of what you should be and shouldn't be doing ("ripping exes from them"), you should rather get them through BNFTP (how game clients get the patches) than get them from someone else over the web. In the context of ease for people to program, it is easier to get the from others or from your game directory after a patch. than to reverse engineer BNUpdate.exe (however it is quite possible).

What are you getting at?

That's what i wanted to say. Reversing blizzard installer doesn't pay off. It's easier to get exes from web (maybe automatically) or from game directory.

[quote author=Ribose link=topic=17663.msg181217#msg181217 date=1232051985]
What are you getting at?
[/quote]

Don't understand  :)

This topic has nothing to do with whether it's a "good idea" or not. It's about how to do it. If one wants an alternative, I already have a system anyone can use to update hashes and verbytes:

Grab the VerByte: http://rcb.realityripple.com/hashes/ver.php?f=STAR (You can replace STAR with any valid BNet game.)
Grab the hashes: http://rcb.realityripple.com/hashes/download.php?f=STAR (Same. This gives you a highly compressed zip file containing the three hash files and the bin file if necessary.)

Since it's been up, I've had new versions available within 6 hours of a release every time, so it's fairly safe to say it's updated. I've already received a DMCA takedown notice from Blizzard, but I'm not taking down these files, as they honestly pose no copyright threat to Blizzard whatsoever.

And someone talks PvPGN were illegal...

I don't believe in laws. Right and wrong are a man-made idea, and legalities are poor enforcements of that deeply flawed viewpoint.

On top of that, I'm not making Blizzard lose money, nor am I causing them harm in any other form.

[quote author=xpeh link=topic=17663.msg181241#msg181241 date=1232068302]
And someone talks PvPGN were illegal...
[/quote]

That's because they are.

[quote author=Ribose link=topic=17663.msg181217#msg181217 date=1232051985]
[quote author=xpeh link=topic=17663.msg181209#msg181209 date=1231992514]
I dont think getting patches from bnet and ripping exes from them is a good idea. It's much easier to download it somewhere from internet.
[/quote]
The game clients download the patches inside MPQ files. If one knew exactly what the client does, they can emulate that just like they do the rest of the Battle.net protocol (Blizzard didn't just give people the source to the clients, people reversed engineered the protocols). In the context of what you should be and shouldn't be doing ("ripping exes from them"), you should rather get them through BNFTP (how game clients get the patches) than get them from someone else over the web. In the context of ease for people to program, it is easier to get the from others or from your game directory after a patch. than to reverse engineer BNUpdate.exe (however it is quite possible).

What are you getting at?
[/quote]

There may never come a day when someone creates a bot that can update itself identical to the way the actual game client does. Technically, it is possible. I don't study the game clients and don't know the very deep details. It would be a lot of work. And quoting Spht from probably 2-3 years ago, "It would be virtually impossible" to create something that always knew how to update itself as the game client does without ever needing a programmer to modify the bot.