Well folks, I'm restarting my bot API from scratch.  Why?  It has some limitations that I'd like to get past.  I just couldn't do it with the current setup.

I'm going for both high flexibility and high performance.  The flexibility is going to come from self-describing connection mechanisms, that can (I'm planning) be as diverse as BNCS, MSN, Y!IM, AIM, IRC, etc.  These can be provided via runtime-loadable assemblies.

I also want to get rid of my incoming and outgoing buffers and just use the types built into the .NET framework -- specifically, MemoryStream and BinaryWriter/BinaryReader.  They'll be encapsulated, perhaps in a derived class.

Another planned addition is priority queues for incoming and outgoing packets -- for instance, a /ban command might get the SystemCritical priority from the command processor, whereas the packet to retrieve the advertisement might get the Low priority.  Alternatively, incoming login events might get High priority where the PING event gets low priority.

At this point, I'm looking for architecture, not feature suggestions.  Code reuse is not a major factor, so be creative if possible :)

Thanks!

If this is just the bot in general and how you're going to code it:  Create a major base class which will not do anything particuarily important for the bot, but perhaps universal error exception functions, variables and structures which keep track of the bot data, etc.  Then from there branch off that class and create maybe 1 - 3 classes that divide the bot up (such as Connection, GUI, UI, etc.).  Then from there branch off as necessary into sub-classes of those classes to make it nice and organized.  Then in your main thread you can polymorphically call all of the functions and access the necessary data and what not and it'll be nice and organized.  You can of course from there, easily expand, in an OO manner, your program to be more complex and powerful and have more features.  The main idea in my suggestion is structuring your program so that each major part of the bot has its own class which derives from a major base class, and then from those classes have many sub-classes that have major parts to do with that part (for instance, the Connection class may have classes PacketBuffer and BnetHashing, etc.).  You will have to play around with the inheritence heirarchy probably, and it may or may not turn out exactly how I described above for whatever reasons.  But give it a try if you want.  It's what I did for my bot and it's great when it comes to updating it since it's so clean and organized.

Question though, what do you mean by yout Bot API?

[quote author=Mephisto link=topic=9242.msg85304#msg85304 date=1098242936]
Question though, what do you mean by yout Bot API?
[/quote]

The mechanisms by which my bot connects to Battle.net, processes data, and handles plugins is what I mean by my API.  The most recently-published version is available here in MSDN-style help format.

Ideally, the connections API and the main, base code that I produce (the BNCS, BNLS, RCRS, local hashing, and BotNet connections) will be encapsulated such that someone who wants to make a bot can pretty much drop in my assembly, derive from a Settings class, hook into some events, and go.

The other stuff is there for additional features.  But yes, that's what I mean by my bot's API.

[quote author=MyndFyre link=topic=9242.msg85299#msg85299 date=1098241153]
I'm going for both high flexibility and high performance.
[/quote]

If you want high performance i suggest using a language that is'nt  interpreted at runtime i.e c/c++, i do agree .net has great  flexibility because im using vb.net/c# and it's excellent but in the end if you want high performance software you are going to have to code it in asm/c/c++.

On the other hand if you like the ease of .net's CLR then maybe consider using mfc it makes coding simple for GUI based apps.

My personal preference is a backend coded in asm/c++ with a mfc front end.

C# and Java really aren't that much slower than compiled code.  Some of Java's libraries (for instance, AWT/Swing, and probably some others) are horribly coded, and run like shit as a result.  But JIT code and interpreted code really aren't that much slower.  I would imagine that C# is much the same as Java in terms of performance.

[quote author=iago link=topic=9242.msg85343#msg85343 date=1098291464]
C# and Java really aren't that much slower than compiled code.  Some of Java's libraries (for instance, AWT/Swing, and probably some others) are horribly coded, and run like shit as a result.  But JIT code and interpreted code really aren't that much slower.  I would imagine that C# is much the same as Java in terms of performance.
[/quote]

Well it's certainly a performance hit translating byte codes to machine code, no one can deny that so i repeat what i said if you want maximum performance you have to write in asm/c/c++ it's that simple.

But with direct machine code, you can't do inline optimizations based on current data.  For example, in C++, if you have a loop that does nothing a million times, it'll take a lot of time.  In Java (and probably in C#), it'll be skipped.  Better performance!

[quote author=iago link=topic=9242.msg85348#msg85348 date=1098295876]
But with direct machine code, you can't do inline optimizations based on current data.  For example, in C++, if you have a loop that does nothing a million times, it'll take a lot of time.  In Java (and probably in C#), it'll be skipped.  Better performance!
[/quote]

A compiler can only be expected to do so much, if you write poor code then it's the programmers fault.

It's not the compilers responsibility to remove poor coding practises, although modern c++ compilers make significant optimizations even to poorly coded work.

Also making optimizations based on current data could you explain further i have little knowlege of this, from what it sounds like i cant see how you can optimize any further than compiling base code as program data is variable i.e it can change rapidly so running a loop one time might not do anything but calling another time might require some sort of complex operation.

[quote author=OnlyMeat link=topic=9242.msg85345#msg85345 date=1098294052]
Well it's certainly a performance hit translating byte codes to machine code, no one can deny that so i repeat what i said if you want maximum performance you have to write in asm/c/c++ it's that simple.
[/quote]
Maximum performance, yes. But if you really don't want to spend 4 times the time to code and debug, you're better off with C#. The performance for this sort of project is negligable, even managed direct3d applications run nearly the same (~5% hit).

[quote author=drivehappy link=topic=9242.msg85353#msg85353 date=1098296878]
[quote author=OnlyMeat link=topic=9242.msg85345#msg85345 date=1098294052]
Well it's certainly a performance hit translating byte codes to machine code, no one can deny that so i repeat what i said if you want maximum performance you have to write in asm/c/c++ it's that simple.
[/quote]
Maximum performance, yes. But if you really don't want to spend 4 times the time to code and debug, you're better off with C#. The performance for this sort of project is negligable, even managed direct3d applications run nearly the same (~5% hit).
[/quote]

So thats why no high end graphics games are written in interpreted languages?

[quote author=OnlyMeat link=topic=9242.msg85338#msg85338 date=1098281458]
If you want high performance i suggest using a language that is'nt  interpreted at runtime i.e c/c++[/quote]

Learn what you're talking about.  .NET code is compiled just-in-time, once.  The native-code assembly is then cached until it is changed; in a release build, it doesn't change frequently.

I'm sorry to come down on you so hard, but I hear that all the time, and people just don't get it.  Java bytecode is interpreted at each pass; .NET code is interpreted once.

[edit] Can a moderator please split off the posts in this thread that are unrelated to this topic -- the topic being an effective architecture? [/edit]

What's [s]BNLS[/s] RCRS?  Or do I have the wrong idea on what it is?

[quote author=MyndFyre link=topic=9242.msg85361#msg85361 date=1098297931]
[quote author=OnlyMeat link=topic=9242.msg85338#msg85338 date=1098281458]
If you want high performance i suggest using a language that is'nt  interpreted at runtime i.e c/c++[/quote]

Learn what you're talking about.  .NET code is compiled just-in-time, once.  The native-code assembly is then cached until it is changed; in a release build, it doesn't change frequently.

I'm sorry to come down on you so hard, but I hear that all the time, and people just don't get it.  Java bytecode is interpreted at each pass; .NET code is interpreted once.

[edit] Can a moderator please split off the posts in this thread that are unrelated to this topic -- the topic being an effective architecture? [/edit]
[/quote]

Dont worry about it you are not coming down on me hard you just agreed with my statement that it's an interpreted language, even if it's compiled JIT once it's still interpreted i.e interpreted  = non native just to keep it simple for you.

And this is also related to your original question in that it's an architectural issue selecting a language based on it's qualities and you also said you wanted fexibility and high performance and i therefore suggested a more appropriate language for high performance applications.

If you dont like the answers dont ask the questions :D

[quote author=UserLoser link=topic=9242.msg85363#msg85363 date=1098298448]
What's [s]BNLS[/s] RCRS? Or do I have the wrong idea on what it is?
[/quote]

RCRS stands for Remote Check Revision Server, its a plaintext server iago wrote for JavaOp and others to use its pretty neat.

[quote author=Warrior link=topic=9242.msg85372#msg85372 date=1098301893]
[quote author=UserLoser link=topic=9242.msg85363#msg85363 date=1098298448]
What's [s]BNLS[/s] RCRS? Or do I have the wrong idea on what it is?
[/quote]

RCRS stands for Remote Check Revision Server, its a plaintext server iago wrote for JavaOp and others to use its pretty neat.
[/quote]

It's an opensource server to do the versioning for logon so you don't need hashfiles or version information locally.  It doesn't (and I refuse to) touch cdkeys and  passwords, because that's just asking for trouble.  It's also decentralized, I have two stable servers, and anybody can run their own, so no chance in them being taken down.

In any case, I don't see what that has to do with this thread.

[quote author=UserLoser link=topic=9242.msg85363#msg85363 date=1098298448]What's [s]BNLS[/s] RCRS?  Or do I have the wrong idea on what it is?[/quote]

It's a Java-based stripped-down BNLS rip-off.  Among its failings:
* Uses text protocol (think http or BNCS chat, rather than bnbinary / BNLS)
* Written in Java (eww!)
* Cannot perform many of the operations for which BNLS is most useful

[quote author=iago link=topic=9242.msg85401#msg85401 date=1098307625]
[quote author=Warrior link=topic=9242.msg85372#msg85372 date=1098301893]
[quote author=UserLoser link=topic=9242.msg85363#msg85363 date=1098298448]
What's [s]BNLS[/s] RCRS? Or do I have the wrong idea on what it is?
[/quote]

RCRS stands for Remote Check Revision Server, its a plaintext server iago wrote for JavaOp and others to use its pretty neat.
[/quote]

It's an opensource server to do the versioning for logon so you don't need hashfiles or version information locally.  It doesn't (and I refuse to) touch cdkeys and  passwords, because that's just asking for trouble.  It's also decentralized, I have two stable servers, and anybody can run their own, so no chance in them being taken down.

In any case, I don't see what that has to do with this thread.
[/quote]
If you have to know the set of servers ahead of time, it's still rather centralized, so I don't think you are truly solving that problem.

Consider that most end users will use the defaults, which provides well-defined points of failure.


MindFyre: I'll give a quick runthrough of how BC4 is being written, in case you want to borrow ideas:

The primary design goals are performance and remote access - what I came up with is...

- A dedicated service that maintains connections to various types of chat networks, and logs events to pass on to clients connecting to the service (with backchat).
- A network-based user interface that does not maintain connections to chat services directly; instead it connects to the dedicated service, receives backchat, and interacts with chat services through the dedicated service.

The important thing here is that the user interface is completely isolated from the backend (user interface operations tend to be very slow compared to most work the backend will be doing).

The service/backend borrows some of it's internal design from BNLS -- there's a single network thread that controls all active connections and maintains state, and a variable number of worker threads (typically but not necessarily one per processor).  Only the network thread directly reads or writes 'state' data, so no locks are required.  All 'expensive' operations are done asynchronously on worker threads (only trivial calculations and state management are done on the network thread).

No blocking operations are permitted in the network thread, as it must be responsive at all times to deal with network events.  DNS operations are performed asynchronously using a dedicated resolver thread (due to a lack of true asynchronous DNS lookup calls in Win32).  All network I/O is overlapped.

Communication between worker-, network- and DNS-resolver- threads is done via I/O completion ports (BNLS used APCs).  I'm not sure what the best analogue of this would be in .NET-land.

BNLS is fairly good proof that the architecture scales reasonably well.

[quote author=Kp link=topic=9242.msg85422#msg85422 date=1098309351]
[quote author=UserLoser link=topic=9242.msg85363#msg85363 date=1098298448]What's [s]BNLS[/s] RCRS? Or do I have the wrong idea on what it is?[/quote]

It's a Java-based stripped-down BNLS rip-off. Among its failings:
* Uses text protocol (think http or BNCS chat, rather than bnbinary / BNLS)
* Written in Java (eww!)
* Cannot perform many of the operations for which BNLS is most useful
[/quote]

I don't see a problem with any of those:
* Sends less bytes than equivilant BNLS.
* It's portable.  We run it on Linux servers.
* Really? I thought the most useful parts were the lack of requiring hashfiles and the lack of having to make changes when the game's version changes.  And those are what I do.  Sending cdkey and password anywhere in plaintext is just stupid, yet BNLS encourages that?

In any case, I wrote it for myself, and I don't care what people think/who uses it.  I never bothered posting here, or on any forum, because it doesn't matter.

[quote author=OnlyMeat link=topic=9242.msg85366#msg85366 date=1098299434]
even if it's compiled JIT once it's still interpreted i.e interpreted  = non native just to keep it simple for you.
[/quote]
Maybe I'm not keeping it simple enough for you.  The JITter compiles the output to native code.  The native code is reused whenever the just-JITted function is called again.  No interpretation necessary.  No JITting necessary.  If I run NGEN.exe on an assembly, it's not JITted at all when it gets to my client.  Learn what you're talking about.

[quote author=OnlyMeat link=topic=9242.msg85366#msg85366 date=1098299434]
And this is also related to your original question in that it's an architectural issue selecting a language based on it's qualities and you also said you wanted fexibility and high performance and i therefore suggested a more appropriate language for high performance applications.
[/quote]
I disagree; language is not part of the architecture, it's part of the requirements.  Often, in the real world, there are reasons for choosing one language or another.  If my specification is that I'm using .NET and also going for performance, then we look at what performance bottlenecks might come up.  If you're working for a business and they want a solution that will integrate with a .NET Web Service on their servers, but they also want other parts to be high-performance, you'll probably choose C#, because the Web Service is already built-in for you, and you can achieve good performance with it.  You wouldn't sit there and argue with them about using C++ because its performance is better, unless you wanted to get fired.

Skywing: thank you.  I'll look at how I might be able to work that into my plans.

[quote author=Kp link=topic=9242.msg85422#msg85422 date=1098309351]
[quote author=UserLoser link=topic=9242.msg85363#msg85363 date=1098298448]What's [s]BNLS[/s] RCRS? Or do I have the wrong idea on what it is?[/quote]

It's a Java-based stripped-down BNLS rip-off. Among its failings:
* Uses text protocol (think http or BNCS chat, rather than bnbinary / BNLS)
* Written in Java (eww!)
* Cannot perform many of the operations for which BNLS is most useful
[/quote]

Of course it can't do many of the things BNLS does, it wasn't designed to - hence the name Remote Check Revision Server. There was another individual who extended some of iago's code and made a BNLS-clone in Java, following the same protocol and such and allowing current BNLS bots to work with it. Only difference is it's open source, written in Java - and, thus, cross platform.

[quote author=Skywing]
DNS operations are performed asynchronously using a dedicated resolver thread (due to a lack of true asynchronous DNS lookup calls in Win32).  All network I/O is overlapped.
[/quote]

I have'nt had a problem with async win32 dns api calls they seem to their job fine why are they not truly async?

I use WSAAsyncGetHostByAddr etc.

[quote author=OnlyMeat link=topic=9242.msg85445#msg85445 date=1098317220]
[quote author=Skywing]
DNS operations are performed asynchronously using a dedicated resolver thread (due to a lack of true asynchronous DNS lookup calls in Win32).  All network I/O is overlapped.
[/quote]

I have'nt had a problem with async win32 dns api calls they seem to their job fine why are they not truly async?

I use WSAAsyncGetHostByAddr etc.
[/quote]

If you look at how those work under the hood, you'll see that ws2_32 creates it's own worker thread and uses (synchronous) gethostbyname, then posts a message to you.

I use I/O completion ports, which are incompatible with window messages (not possible to wait for both a window message and a completion packet at the same time in the same thread).

Additionally, the various WSAAsyncGetXByY functions don't support IPv6 -- I have native IPv6 support everywhere, which requires support for hostnames with IPv6 addresses associated.

MSDN also recommends against using WSAAsyncGetXByY:

[quote]WSAAsyncGetHostByName
The WSAAsyncGetHostByName function asynchronously retrieves host information that corresponds to a host name.

Note  The WSAAsyncGetHostByName function is not designed to provide parallel resolution of several names. Therefore, applications that issue several requests should not expect them to be executed concurrently. Alternatively, applications can start another thread and use the getaddrinfo function to resolve names in an IP-version agnostic manner. Developers creating Windows Sockets 2 applications are urged to use the getaddrinfo function to enable smooth transition to IPv6 compatibility.
[/quote]

[quote author=MyndFyre link=topic=9242.msg85442#msg85442 date=1098315392]
[quote author=OnlyMeat link=topic=9242.msg85366#msg85366 date=1098299434]
even if it's compiled JIT once it's still interpreted i.e interpreted  = non native just to keep it simple for you.
[/quote]
Maybe I'm not keeping it simple enough for you.  The JITter compiles the output to native code.  The native code is reused whenever the just-JITted function is called again.  No interpretation necessary.  No JITting necessary.  If I run NGEN.exe on an assembly, it's not JITted at all when it gets to my client.  Learn what you're talking about.
[/quote]

Yes im not a .net expert but i do know how the JIT process works in .net, the JIT occurs every first time a method is run in each session which means caching only takes place during the application lifetime, as soon as you run the application again the same process occurs BANG performance hit after performance hit. Therefore it's effectively interpreted everytime it's run ( per method ) when the MSIL is translated to native. Learn what you are talking about.

[quote author=MyndFyre link=topic=9242.msg85442#msg85442 date=1098315392]
[quote author=OnlyMeat link=topic=9242.msg85366#msg85366 date=1098299434]
And this is also related to your original question in that it's an architectural issue selecting a language based on it's qualities and you also said you wanted fexibility and high performance and i therefore suggested a more appropriate language for high performance applications.
[/quote]
I disagree; language is not part of the architecture, it's part of the requirements.  Often, in the real world, there are reasons for choosing one language or another.  If my specification is that I'm using .NET and also going for performance, then we look at what performance bottlenecks might come up.  If you're working for a business and they want a solution that will integrate with a .NET Web Service on their servers, but they also want other parts to be high-performance, you'll probably choose C#, because the Web Service is already built-in for you, and you can achieve good performance with it.  You wouldn't sit there and argue with them about using C++ because its performance is better, unless you wanted to get fired.
[/quote]

I disagree i think certain architectures are not possible in all languages for instance i was working on a software contract developing a generic data transformation for various file formats, the original language was going to be vb 6.0, but after much discussion it was decided vb's threading model was not appropriate for the transformation of multiple formats simitanously therefore a new language was selected with a more efficient threading model - 'C++'.

As a contract software developer working in the 'real world' i have practical experience of these issues. of course solutions can be developed completely independently of the  software language but when the need arises for specific performance, scalability and other qualities then certain languages meet these requirements better than others. and this is infact an architectural issue.

Hey, kind of getting back to the original topic, I think I'll throw in my two cents, as I thought about a project like this previously.

Basically what you would like to do, I take it, is provide good enough abstraction that your app can be used for chat and IM protocols.  Even within the two chat protocols you're considering (BNCS & IRC), there are some fundamental differences.  On Battle.net, for example, you can only be in one channel at any given time, but this doesn't hold true for IRC (if I recall... I never used IRC much).  Not to mention that the general auth/login procedures are very different.  You'll need some sort of mechanism(s) for describing the limitations of each particular protocol.

Then you have IM thrown into the mix.  While all the IM services you listed support group chat, these are different in that (again, IIRC, YMMV, etc.) you normally don't create a globally accessible "channel", but rather invite people to conversations.  Although, if you treat every chat channel and IM conversation as the same type of "channel", headaches can be reduced.

And as far as commands go, you'll run into trouble with those between services.  Unless, of course, you provide methods to create those commands (e.g., ban on BNCS/IRC will run a /ban, but on, say, MSN, will block the user).

If it were my project, I would have a "Service" class that all chat/IM services would derive from; possibly generic Chat and IM classes that the services would derive from instead.  Conversations - regardless of whether they are actually channels or IM convos - would all be represented by the same class.  (I'm of course assuming you're thinking OOP, since you're using .NET.)

But I'm pretty new to all this, and I'm sure you can come up with something better.  I am interested in how you actually implement it, though.