To: stik@ON-Luebeck.DE
Subject: Re: [10] STIK: STIK/DIALER                                                       

On Thu, 22 Aug 1996 22:33:00 Peter Rottengatter <perot wrote: 
>
>On Thu, 22 Aug 1996, Ronald Andersson wrote:
>> 
>> Ok, I have done so, and will now add some answers below.
>> Now, you read all of those before responding too.
>
>I've done so. A few misunderstandings arose, which might be because I had 
>to hurry when I wrote that message, so some of my wording was rather 
>unprecise. Sorry for that.

Ok, that happens to everyone occasionally, not least because we often discuss
complex features not even existing yet except as vague implementation plans.


>> >Naah, cannot understand this. The calls to receive data do not block.
>> >If there is no data they return with a corresponding 'error'.
>> 
>> I am not a client programmer (yet), so I am not fully conversant with the transfer
>> protocol, but I have seen comments even from Dan about the 'blocking' effect.
>> If this is merely due to a misuse of the function then all client authors should
>> correct their code immediately.
>
>That's what I meant, by saying I haven't written any data traffic code yet.
>I was argueing from the specs, which I examined thoroughly, and some 
>early message, can't remember the author, saying that he went trough all 
>the code for figuring out why transfer was slow, only finding TCP_Open 
>as the only call that would block for a short moment. If there is this 
>blocking effect on receiving data, then you`re right, and I've written 
>utter nonsense ;-)

Since Dan has now confirmed the non-blocking of these functions, there is no more
to be said about that, except that client authors should make better use of it.


>In that situation I would strongly recommend to work on the blocking to 
>remove it, rather than inventing other interfaces.

For that single purpose I agree, we should certainly use the means already at our
disposal to improve existing clients as far as this allows.
But there are other aspects of multi-tasking multi-client use needing attention.

Again I'm forced to emphasize that whatever changes needed should be made as a
complement to the already existing interface, not as a replacement for it.
As for 'inventing', what could possibly be wrong with that...?


>There is a significant consequence if we figure we could do without AES 
>messages : STiK doesn't have to be a GEM program, it could get completely 
>installed from the auto folder, and would free an accessory slot. It 
>would make installation simpler, and, last but maybe not least, still be 
>accessible by pure TOS programs.

Since I have stated repeatedly that the old interface _must_ remain, even though
it may need some modernisation for multi-client use, the interface as proposed by
me would still be accessible to _all_ programs, though it might have routines in
it that would only be needed by GEM applications.

I do not, however, see any need for any TOS-type or TTP-type clients, and today
no such clients exist AFAIK that have not been replaced by GEM applications.


>> NB: This requires that STiK actually starts processing of a reception packet for
>>     this client after releasing him for other things with a 'data_not_yet_ready'
>>     type of error code, so that client can call back later.
>
>It'd be perfectly possible to return DATA_NOT_THERE, if there is no 
>(raw) data from the serial port. As soon as there is data it would not 
>return but take a little time to process it, then decide whether the 
>packet is for the calling client and return either DATA_NOT_THERE or OK.
>This is not difficult to implement, so I guessed STiK would have it, and 
>bearing in mind that old message I mentioned above I'd certainly expect 
>STiK to behave this way. This would enable, say, redraws. However I keep 
>observing the clients not doing anything even though no data is pouring 
>in (to be seen from the modems LEDs). It must be either in STiK or the 
>client where the waiting takes place. I'm not perfectly sure, but up to 
>now I thought it would be on the clients side (calling the receive 
>function in a loop without evnt_multi() until all data is transfered).

That appears to be the case, according to Dan's response, and this should of
course be amended ASAP in all of those clients, but this will only get us part
of the way to a fully non-blocking multi-client implementation.


With a truly non-blocking approach, packets will normally be completed between
calls of the clients, which a multi-client STiK version would handle by moving
the packet into the individual buffer of the client and updating some flags in
the individual client structure containing that buffer.

If something goes wrong with that client, it will not affect the master buffer
of STiK or any operations of the other clients.


>> You do not mention the transmit routines above, so what about them...?
>
>What I'm saying is that the actual traffic routines take *some* time, but 
>they wouldn't need to lock up the machine completely. Take an MTU of 1500 
>as a common example, then with a 28800 bps modem an IP packet would need 
>half a second to get transmitted or received. Thus assuming STiK would do 
>transceiving one packet en block, any client could react to user 
>interaction with a dead time of at most half a second. Current reality 
>obviously is far worse.

Yes, and even half a second is too much in a multi-client context, where we also
have to assume incoming packets may be for _other_ clients than the calling one.

Using multi-client approach for block transmission, would allow each client to
have packets awaiting transmission without blocking any of them, except when
further operations require knowing the network response, and even that would
still leave them active in their event_loop while waiting.

Again, errors caused by one client should affect no others or STiK itself.


>Receiving data would probably work always block by block. TCP_send can do 
>differently. If you provide 20 K to transmit, this would be 14 blocks, 
>thus should take seven seconds during which the client is frozen. If you 
>do it in small chunks, say, smaller than MTU then TCP_send should return 
>immediately, if the packet was not filled up, or take half a second to 
>send the full packet. Again half a second dead time at most.

There is no reason for freeze time to have any connection with data size,
which only should affect the time taken to transmit the packets.  That is
an operation in which the client takes no active part, so it should be
released to await completion of the operation in its event loop.


>Half a second is still not smooth enough. I guess the only way around 
>this is exchanging the polled ports by interupt driven ports. I've been 
>thinking about this. Possibly the whole thing could be done by modifying 
>the device-lib STiK uses for port traffic. That way Dan wouldn't need to 
>spend any time on this. It must be done by someone with solid assembler 
>experience. I could do it, but I'm short of time, and it would be hard to 
>do any more debugging on the Dialer ;-)

Well, I'm not well informed on the device lib, but with sufficient info
I should be able to do it too, but I suspect many STiK routines need to
be part of the actual interrupt routines and don't know if it's practical.

It might be better to use a routine linked to the 200Hz system interrupt
instead, but that should of course be in speed-optimized assembler.

I'm perfectly willing to take this on when we are certain that the CSLIP
routines are as functional as the SLIP ones. I have _no_ feedback on this yet,
except from myself and Dan, for whom they now appear to function well.
(I've stayed in CSLIP for several days now.)


>> Let me repeat again: I have _never_ said that transfer handshaking is to be done
>> by AES messages.  For this purpose I (like you) prefer other _faster_ means.
>> I just don't think the current means are sufficient, though this may be partly
>> due to misuse or non-use of existing features (but only partly).
>
>Misunderstanding : Here I was talking not mainly to you, Ronald. I know 
>meanwhile how you meant it.

Good!  Actually it may be a good thing if we occasionally overemphasize the
'need_for_speed' so that nobody misses it.  In the 'high-level' world of
programming speed-optimization is apparently a dying artform.
(eg: Windows)


>I'd state it this way : The clients could be far more interactive if the 
>calls for receiving and transmitting are (were ?) not blocking. The best 
>behaviour we would get by having the transceiving purely in the 
>background, done by interupts.

I agree completely, and since this requires a rewrite to get new interrupt
routines, we should take this opportunity to add the missing pieces for full
multi-client support.


>> In fact Alexander himself has complained that the current interface limits smooth
>> functioning of CAB, so I think he would welcome an improvement of this nature.
>
>Did he ? Of course you're right. To me it sounded like some people (not you !) 
>would think a AES message driven STiK would solve every problem, including Life,
>Universe and Everything ;-). I was just pointing out that without lots of
>modifications on the clients we would gain little.

Naturally any improvements to STiK are useless if unused by the clients,
and for future full multi-client support these changes may be extensive.
So the old-style interface should remain, allowing gradual changes.

As for the larger question, I thought everyone new the answer...:

Sorok-dva, Gospodin.


>I didn`t know that the interface has been altered for mailto:. This means 
>the CAB dev-kit I've got is worthless now ;-)

What's a CAB dev-kit ?  Do you mean the old CAB.OVL source code...?
Well, it's incomplete rather, since I expect other things would be unchanged.


>> No, I don't think so, even if all you say above is true, that still leaves several
>> unanswered questions, such as transmit operations and simultaneous clients.
>
>Simultaneous clients are a completely different matter, I think. For this 
>the problem of reentrancy has to be reexamined again. If the calls() are 
>properly reentrant then simultaneous clients would work.

Yes, that was part of my point in bringing up the need for new individual buffer
structures supplied by the clients.  Any data of STiK affected by a client must
either be individual to that client, or intended to have global effects.


>> >CAB however (sorry that I gotta rob your illusions ;-) )wouldn't benefit anyway,
>> >unless Alex gives up on separating the net code from the browser code.
>> 
>> Where did you get that idea.  All it takes is for him and Dan to agree on an
>> altered interface to CAB.OVL, so that Alex can have a special event_loop handling
>> things while waiting for transfer completion.
>
>Sure, you`re right, see above. I was just trying to be provocative.

Well, that might have been a good idea too, except that Alex doesn't read this
list, so we must write directly to him to get him to do anything.


>> I have _never_ dismissed the current interface, only suggested added improvements,
>> but your comments on sequential programming are of course true.
>> 
>> As I've understood some of Dan's remarks one problem of the current interface is
>> that clients are considered purely sequential and alone, whereas we need them to
>> work as parts of a multitasked environment (even under single-TOS).
>
>I can't see whether this affects the actual _interface_. However 
>limitations might arise from the implementation on STiKs side. I guess 
>it's this what you meant. Or do you wanna say there is a problem with the 
>interface itself ?

No, it's mainly such things as lacking interrupt structure and separate client
data structures that mars the current implementation.


>I was just trying to reconstruct how the idea evolved. I've been added 
>onto the mailing list later.

I'm not one of the 'old-timers' either, but you may still be correct about how
the idea of AES messages for 'everything' evolved.  This should cause no
argument between us two though, since neither of us believe in this idea.


>I talked to more people than just you. After all, this is a mailing 
>list, and my message was not private e-mail that only you could read. 

Actually I got a little worried that others would start to think I was for
those ideas I've been arguing against, causing lots of confusion.


>Believe it or not, but I noted far earlier that you're not favouring data 
>transfer by messages. Lets finish this subject.

Good, I agree.


>> >But remember that the current STiK already provides events
>> >(at least you can look at it this way),
>> >it's just that these events are not delivered by event_multi(),
>> 
>> In principle I agree, but even Dan thinks the current implementation insufficient,
>> and I agree because we need better multi-client support, and a consistent way of
>> letting clients stay non-blocking in their event_loops (for both RX and TX).
>
>I agree perfectly. We just disagree on how to achieve this. I think AES 
>messages are simply not the right way, whether for data transfer or for 
>other things.

Actually we agree on most things, once all misunderstandings are clarified.
I also see no absolute need of using AES messages in STiK at this time,
but I do want STiK to retain the ability to use _all_ TOS functions.


>I've never read Dan saying anything about this. Are you citing private 
>e-mail traffic ?

Not exactly quoting, but he has mentioned the need for improvements, yes.


>> And what about other calls of the interface...?  (none must block)
>> And what about simultaneous (well nearly) client calls...?
>> This is where the current interface fails to do the job.
>
>The *interface* or it's *implementation* ? That's a difference.

Agreed, though I think it's better to add new functions rather than changing
old ones to incompatible usages.  This way adaption to the new methods may be
done gradually rather than in one all encompassing step (harder to debug).
All multi-client functions needing new buffers will require new arguments.


>> >Ok, I'll sketch a rough example for the event_loop :
>> ----- snip -----
>> >Hopefully you get the point ?
>> 
>> I sure do, since this is one of the forms of "event_driven" uses of the interface
>> which I favor, but the current routines need alterations for several reasons,
>> since some _are_ blocking, and most lack multi-client support.
>
>Good, we have the same opinion on this !!! So lets discuss how to remove 
>the blocking, and we've done the first step.

Yes, and the very first step is thus complete, since Dan has confirmed your
opinion on proper use of the transfer routines.


>> No, that is a common flag for all clients, whereas I speak of a unique flag
>> for each client, and therefore supplied by the client.
>
>This flag must be connection based, not client based. If a client has two 
>connections open then one should not block the other only because it's 
>got no data to pass.

That is quite correct, and was what I actually meant.  Clients must supply
separate flags and buffers for each separate connection to be used.

>This must be there already, I guess otherwise applications having two
>TCP connections open at the same time (like ftp) won't work at all.

Yes they will, because the current routines can keep track of several packets at
the same time, having different recipient clients.
But AFAIK overflow (ignored data) of one client may affect all.


>> >The only trouble is that you can get buffer overflow if the application spends
>> >to much time doing other things, that's why I proposed a set_buffer() call in
>> >STiK, so that the application can have a larger buffer.
>> 
>> Which I agree with, except that each client should have separate buffers.
>> STiK still needs a master buffer, but packets should be transferred to the buffer
>> of the client that ordered the data.  This way overflow only affects the buffer
>> of the misbehaving client, and not the master buffer or other clients.
>
>Sorry, I was unprecise again. Of course set_buffer() must set the 
>_application_s buffer. I think not the actual packets should be 
>transferred from a master buffer to a accs buffer, but (GPKT *) pointer. 
>It really wastes to much time to transfer the actual data.

But unless that is done there is no protection against overflow.
One client who has stopped reading its data could thus prevent other clients from
getting any, because the global buffer fills up and is never emptied.

That will then have to be solved by throwing away all packets addressed to that
connection, including those already received, and setting a flag in the buffer
structure for this connection, so the client can see what has happened if he
wakes up later.

Hmmm, yes that might work acceptably too I think.
But here it's my turn to remind you that we really need one buffer per connection,
rather than one per application, of course.


>> >This sounds reasonable. But don't you think the semaphore is there already ?
>> >STiK tells you if data is present.
>> 
>> Present, but insufficient for multi-client use.
>
>Note that I was thinking about a different semaphore. What's your opinion 
>on that one ?

Please clarify.
A simple 'data_is_present' flag can not discriminate between connections.
Each connection should have a separate flag telling the client whether he has
data for that connection or not.


>> >> For multi-client use STiK can then have a:
>> >> 
>> >> 'struct client_blk *client_list[ MAX_CLIENTS ];'
>> >> 
>> >> or something similar to keep track of all currently active clients.
>
>I'm not all to happy with keeping tracks of clients. I used to compare 
>TCP connections with HD files. If process a opens files 1 and 2, and 
>process b opens file 3, then Gemdos doesn't care that two of the files 
>belong to process a. For Gemdos there are just three files open. You can 
>even exchange handles, so that afterwards process b works on file 1, say.
>Similarly STiK shouldn't care if there is one or two clients working, it 
>just should keep track of connections, like Gemdos in my example.

Correct!  Let me rephrase the above example then.

struct conn_blk *conn_list[ MAX_CONNS ];  /* keeps track of connections */

But even so, I find it higly suspect to allow  two clients to work through the
same connection, though I agree that this need not directly concern STiK.


>What's your opinion ? Why do you wanna keep track of clients ?

No, that is irrelevant to STiK internals, I really meant connections.
The problem is more obvious in multi-client use, which caused me to say 'client'.


>> AFAIK the CAB.OVL is still supported by Dan, and I'm sure he could do something
>> to alter the interface so that CAB can stay in the event-loop awaiting data.
>> And if your description is indeed correct and reception calls never block then
>> this should be possible immediately without waiting for STiK changes.
>
>Now you know what I meant, and you see that we basically agree on all 
>this, I guess ??

Yep, we sure do.


>> But why then have none used it yet, and why do they say that the calls block ?
>
>I have no idea, and it puzzles me even more because I actually *read* 
>people saying that the traffic call() won't block.
>
>
>> Are they in fact blocking or not...?
>
>To block, or not to block, that is here the question ... ;-)

No longer !
Dan has confirmed your view of this !

-------------------------------------------------------------------------
Regards:  Ronald Andersson                     mailto:dlanor@oden.se
                                               http://www.oden.se/~dlanor
-------------------------------------------------------------------------
