Okay...

I'm working on a set of network connection objects that handle incoming and outgoing data processing in a variety of ways that can be extended.  What I'm primarily concerned with is applying an arbitrary number of filters, such as compression / encryption algorithms, to a data stream.  I don't want this to be required, however, for connections such as a BNLS connection.

This is what I have so far, and I'd like your opinions.

ConnectionBase class
--Handles a single TCP/IP end-to-end connection in either direction.
+ OnDataSending( ) : void
    --Called when data is about to actually be sent -- that is, it was dequeued (if it was enqueued previously) or was immediately sent.  Upon method completion (callbacks called, etc.), this method actually passes the data onto the socket.  This method can be cancelled via a parameter.
+ OnDataSent( ) : void
    --This method is called back when the asynchronous "send" operation has completed.  It contains the data that was sent as a parameter.
+ OnDataReceiving( ) : void
    --This method is called back when the asynchronous "receive" operation has completed with data read from the network.  It contains the data that was received as a parameter.  This method can be cancelled via a parameter.
+ OnDataReceived( ) : void
    --This method is called with the received data buffer as a parameter.  Upon completion of this method, the data is processed.

FilteredConnection class : Derives from ConnectionBase
+ OnDataSending( ) : void
    --Notifies callbacks with the unfiltered data.  Applies filters to the data. Initiates the data transfer with filtered data.
+ OnDataSent( ) : void
    --Method is called with data on the socket callback.  Parameter's data is filtered.
+ OnDataReceiving( ) : void
    --Method is called with data on socket callback.  Parameter's data is filtered.
+ OnDataReceived( ) : void
    -- Data is unfiltered. Callbacks are notified with the unfiltered data.

Am I being redundant?  This design doesn't "feel" good.

Regarding OnDataSending - what if a filter wants to modify the data, but needs more information before it's able to decide what the output will be?

[quote author=Adron link=topic=11403.msg110144#msg110144 date=1114631747]
Regarding OnDataSending - what if a filter wants to modify the data, but needs more information before it's able to decide what the output will be?
[/quote]

Like what?

I can think of a few filter scenarios:

1.) Data compression.
2.) Data encryption.
3.) Blocking of certain data from being transferred.

The first two are negotiated at the very beginning of the connection.  Generally, the connection negotiates version and then filter settings, and then applies filters to all communication that follows.

The third scenario is something like iago had in JavaOp -- I can optionally apply it to -- say -- a Battle.net connection, and it checks to see if my CD key or password is being sent, and then if it is, changing the bytes in the buffer to be sent.  If the filter is being applied to the connection, it will already have the data (such as CD key and password) available to it in the connection configuration object.

So I guess my question is -- what do you mean?

Example 1: You have a block mode cipher working with a block size of 8 bytes. You just received 7 bytes, but you need one more before you can get to work.

Example 2: You are filtering b.net packets. You just received four bytes, "ff 0f 21 00", but before you process them you need to look at the rest of the packet.

For example 1: the header of the protocol I am designing specifies the length of data contained in the message.  The OnDataReceiving() method won't be called until all of the data comes in.  So, there is not necessarily a 1-to-1 correspondence between the actual callbacks and the OnDataReceiving() method calls.  But the callbacks are not visible to outside or descendent classes (they are private).

For example 2: well, I really can't think of an instance where I would be filtering incoming packets, necessarily.  But the same instance applies as in example 1.  Battle.net packets wouldn't be processed until the entire payload is received, and that is part of the BNCS message header.

[quote author=MyndFyre link=topic=11403.msg110188#msg110188 date=1114642205]
The OnDataReceiving() method won't be called until all of the data comes in. 
[/quote]

That sounds bad. These kinds of classes should be generic, so that they are reusable. To wait for all the data to come in requires that you know when all the data is there, and that is protocol-specific.

This situation sounds like a job for the Chain of Responsibility pattern:

[quote]
Avoid coupling the sender of a request to its receiver by giving more than one object a chance to handle
the request. Chain the receiving objects and pass the request along the chain until an object handles it.

Use Chain of Responsibility when:

- more than one object may handle a request, and the handler isn't known a priori. The handler
should be ascertained automatically.
- you want to issue a request to one of several objects without specifying the receiver explicitly.
- the set of objects that can handle a request should be specified dynamically.
[/quote]

You could have the handler as the last object in the chain. Objects that are responsible for decompressing or decrypting would have the opportunity to do that before the message reaches the handler. You could also write these classes such that objects in the chain can say "I need more data to be able to process this", in which case, the network IO class can buffer whatever has been received and send the old data along with any new data that arrives, until one of the objects in the chain reports that they received enough data to be able to process something. This pattern has excellent reusability/extensibility, because new functionality can often be created simply by adding an object to the chain.

I'm sure Google will yield more information on the pattern.

Uh.

See the Java streams API. It's what you want. In particular, InputStream, OutputStream, and take a look at some FilterInput/OutputStreams.

[quote author=Banana fanna fo fanna link=topic=11403.msg110218#msg110218 date=1114648975]
Uh.

See the Java streams API. It's what you want. In particular, InputStream, OutputStream, and take a look at some FilterInput/OutputStreams.
[/quote]

I'm not using Java.  :P

Arta: Thanks for the info.  I'll look that up.  I've not heard of the design pattern you're talking about, but it sounds interesting.

[quote author=Arta[vL] link=topic=11403.msg110206#msg110206 date=1114647735]
the network IO class can buffer whatever has been received and send the old data along with any new data that arrives, until one of the objects in the chain reports that they received enough data to be able to process something.
[/quote]

From this, it sounds like if one object in the chain doesn't have enough data to process, the data should be passed on to other objects to see if it's enough data for them. That's probably not really what you want.

Perhaps not - maybe a "queue of responsibility" would be better :)

[quote author=Arta[vL] link=topic=11403.msg110403#msg110403 date=1114779969]
Perhaps not - maybe a "queue of responsibility" would be better :)
[/quote]

Especially if object 1 decrypts the data and object 2 processes it :)

So, it seems I should decouple the connection handling code from any type of data processing, and have prioritized handling objects?

In your opinion, should the handling objects specify their own default or suggested priority, or demand a priority, or be user-configurable?

Okay, I've generated a generic schematic of how a chain of responsibility would work in my situation (forgive me for the yucky Word-illustrated pictures):

Incoming:
[img]http://www.jinxbot.net/Samples/cor-incoming-data.gif[/img]

Outgoing:
[img]http://www.jinxbot.net/Samples/cor-outgoing-data.gif[/img]

Doing so has raised a few questions in my mind...

1.) In my mind, a "Need More Data" request in the incoming chain makes sense, since you can treat it as a pull model.  But in the past, I've spawned new threads to parse data; for example, my incoming request would do something like this:
--Data Received callback function (thread A)
---Store all data in a buffer.
---Start async receive method
---Figure out how long the data was supposed to be.
---If it is long enough, start Thread B to process it.
---If it is not long enough, store the buffer and wait to get the rest of the data.

Now, however, it seems I can't start thread B because Thread A might need to give Thread B more data.  Is this accurate?  Note that in the past, my connection objects were aware of the protocols they implemented.

2.) Also, the "Need more data" idea seems to not make sense in a push-type function such as when the data being sent.  It doesn't really make sense that the RSP service would tell my client's BNCS handler "I need more data to be able to send to the server," because the BNCS client is actually pushing the data to the server.

The idea of passing on to other objects is fine, except that for certain filters such as compression and encryption, I believe that the order in which they are applied is important.  The server and client negotiate these ahead of time, and they are applied in the order in which they are negotiated.

RSP? .. RBP?

[quote author=UserLoser link=topic=11403.msg111003#msg111003 date=1115173986]
RSP? .. RBP?
[/quote]

RSP is an RBP-like protocol that does somewhat more.  It is preliminarily documented here and when I finish the documentation I was going to be asking for comments here.

I don't think that 'need more data' will happen for outgoing messages. When you've decided to send something, you don't really want that send to be delayed because something needs more data. I can't imagine why an object  in the outgoing chain would need more data anyway?

Your incoming chain looks good, but under that model you'll have to make sure that the objects in the chain are ordered according to the amount of data they need before they can process something. I'm not sure that it's necessary to have that eventuality for all your objects, either: isn't it really only the final BNCS stage that gets to say "this is an incomplete message"? Why do the other objects have length requirements?

Thread B would certainly have to wait until thread A had provided enough data before it could process it. Why are you using a separate thread for processing, though? I'd do it all on one thread, or perhaps use a pool of worker threads - I think the latter is overkill for a client application, though, unless it's going to be doing an extraordinary amount of work.

It may happen if you need to filter outgoing b.net packets and get only the header first....

Why would you only get the header first? I'd think that would be an indicator of bad design.

[quote author=Arta[vL] link=topic=11403.msg111039#msg111039 date=1115221477]
Why would you only get the header first? I'd think that would be an indicator of bad design.
[/quote]

This is going to be a server application.  ;-)

Thread B was there so that my looping mechanism didn't hold up the rest of the application.  But it was only used when I implemented my connection object knowing in advance the type of protocol I was using.

Also -- Adron suggested that I might use some type of encryption plugin that required a set number of bytes before it could process the message.  That number could easily be defined in the resultant filtered data.

I meant you should look at the java streams API for design decisions.

[quote author=Arta[vL] link=topic=11403.msg111039#msg111039 date=1115221477]
Why would you only get the header first? I'd think that would be an indicator of bad design.
[/quote]

If this is an easy-to-use generic network connection, that people can use to make bots, I could see one person writing a filter to do something useful, and some newbie writing a bot that sends one byte at a time, and those being used at the same time.

Hmm, yes, which would be bad design :)

In a highly OO system, outgoing messages should be encapsulated by some object. It shouldn't ever be possible to send bits of messages - which, in a command chain, should be atomic - separately. The incoming chain is a different matter, obviously.

[quote author=Arta[vL] link=topic=11403.msg111552#msg111552 date=1115574951]
Hmm, yes, which would be bad design :)

In a highly OO system, outgoing messages should be encapsulated by some object. It shouldn't ever be possible to send bits of messages - which, in a command chain, should be atomic - separately. The incoming chain is a different matter, obviously.
[/quote]

That's true, but the way I understood this system, it was meant to generically encapsulate a tcp stream. In a tcp stream, the minimum message is a byte. And if some part of the chain wants to process more than one byte at a time, it needs to be given the opportunity.

Realistically, I wanted to be able to expose both methods.  I wanted to make it so that derived classes would be able to get the byte streams, but classes utilizing the derived class would simply receive the "message" objects.  I've used a similar design before, except that in my old classes, the derived classes controlled their own sockets.  This time, I want the base (even abstract) class to control the socket and provide a nice interface to derived classes.  But I don't know that it's possible.