I am *attempting* to write a a class that inherits from stream that encapsulates many files into a single stream. Because of the nature of the program I am writing (bittorrent client), it has to be asynchronous(sp?). The writing part should be easy enough, something like the syncro method I wrote, but just calling BeginWrite because there is not any data that needs to be handled for it. But for the read, the data read must be processed as soon as it is read. The sync method I wrote looks like (only changed Write() to Read() and some other modification, not commented though):

[code]
private void _write(byte[] buffer)
        {
            int start_position = position;
            int end_position = position + buffer.Length;

            int start_index = 0;
            int end_index = 0;

            int buffer_position = 0;
            //find the first file to open
            start_index = GetIndexOfFile(start_position);
            //find the last file to open
            end_index = GetIndexOfFile(end_position);

            //-------------------------------------------------------

            //Start writing to files
            for (int i = start_index; i <= end_index; i++)
            {
                OpenFile(i);
                int max_to_write_to_this_file = file_info[i].EndIndex - position;
                int position_in_file = position - file_info[i].StartIndex;
                //if the max to write is larger than the remaining length of the data,
                //the last position is the end index less the remaining length of the data,
                //otherwise it is the end of the file
                int last_position_in_file = (max_to_write_to_this_file > buffer.Length - buffer_position) ?
                    file_info[i].EndIndex - (buffer.Length - buffer_position) :
                    file_info[i].EndIndex;
                int count_to_write = last_position_in_file - position_in_file;
                file_streams[i].Write(buffer, buffer_position, count_to_write);
                position += count_to_write;
                buffer_position += count_to_write;
                CloseFile(i);
            }
        }
[/code]
(Critizism(sp?) wanted)

I can't think of a clean way to implement this async-ly.

Bump!

I still haven't implemented anything, but I've had a thought.

Implement some sort of queue system where each item in the queue has 'work' to do. Each item could have a 'subsequent' item, one that gets put in the queue once that item has finished. So item A would be read the data, then put B in the queue. B would be process the data, and B could have C, and so forth.

Anyone have any thoughts on this?

[quote author=Shout link=topic=15396.msg161387#msg161387 date=1163921598]
Bump!

I still haven't implemented anything, but I've had a thought.

Implement some sort of queue system where each item in the queue has 'work' to do. Each item could have a 'subsequent' item, one that gets put in the queue once that item has finished. So item A would be read the data, then put B in the queue. B would be process the data, and B could have C, and so forth.

Anyone have any thoughts on this?
[/quote]

Looks like a bastard of queuing and threading to me

Edit: Asked around, apparently its a 'parallel queuing system'.

Hrm.

BitTorrent should support multithreading out of the box because a BT client shouldn't overwrite sections of a file because you shouldn't be downloading the same section of a file from multiple remote locations.

I'd suggest the queuing system you considered.  One thread manages disk I/O while your sockets receive the data and push it by block to the disk I/O thread.  There can be n threads performing network I/O that way, and it also simplifies updating your user interface.