Hi Vassilis,

as you know, this has been anything but a quiet list lately, so I've had to
prioritize what to answer at once.  Your mail is over 10 days old now, and
some things in it are no longer relevant, or already agreed, so I'll be
'snipping' those sections (in some places without remarking on it).


On Wed, 25 Nov 1998, Vassilis Papathanassiou wrote:
>
----- snip ----- re: using MagiC features for STinG
>
>Sorry, but I (being lazy ;-) thought we could pass the ball to MagiC's
>field ;-) at least for some functions that I felt they could benefit from
>that (personal opinion of course).

That is a valid solution, although specialized for the MagiC environment.
In developing for multiple environments however, it can be very 'expensive'
in terms of work effort, to use separate methods for each environment.
Own methods that work with all the environments save on that effort, thus
liberating more programming time for further developments.


----- snip ----- re: more on various MagiC features
>
>This was because I felt that the time (inside Peter's studies) was not
>appropriate for such discussions. But I pointed to DEV_LPT1 as an
>example, so that anyone could study it a little, and even come with the
>conclusion that it can't be done ! (because after all I don't know the
>exact nature of STinG's interrupt scheme).

STinG itself only takes direct part in one interrupt.  This is the system
clock interrupt, running at 200 Hz, giving a 5ms timeslices to STinG as
well as to other system components. This interrupt is the 'heart' of low
level TOS and must never stop beating, so it must be properly implemented
or at least emulated in ALL the TOS compatible environments and emulators.

That is the reason why it was chosen to also be the 'heart' of STinG, and
thereby give STinG automatic compatibility with all TOS compatible systems.


>Fortunately STinG has a cookie and so does MagiC, so with the same STING.PRG
>STXs etc, they could talk to each other (I mean without splitting, just a
>STING.DEV thrown into \gemsys\magic\xtension). This could be growing (in size
>and capabilities) even without been noticed by us (client authors), until the
>date that Peter would decide to 'export' some capabilities to the API.

That could be done, but it would be best if some common method could be used
internally, for all systems.  Then very small specialized drivers could be
used (if still needed) to link them to the specific system in use.  In fact
I've been thinking of producing an XFS for BetaDOS based on this idea, so
that a small XFS could link in BetaDOS drives into MagiC's U:.  The problem
is to keep that XFS small and referring to the BetaDOS kernel in such a way
that the XFS won't need frequent updates (thus stealing development time).
At present I lack both time and XFS-writing experience to do that, so I'll
leave it as an unrealized idea for the present.


----- snip ----- re: suggestions for non-blocking TCP_close and Resolve calls
>
>And which (both) I found not only good, but also easy to implement (personal
>opinion again, to prevent misunderstandings on what is 'easy' and what not ;-)

The implementation would be quite easy compared to some things already done,
such as making STinG a natural and reliable part of the system interrupt...
That was rather tricky and required intense thinking by both Peter and me.

The new API ideas are mainly based on the tried and true principle of GEM,
that each calling program should provide its own arrays for input and output
data, thus eliminating most reentrancy problems even under multitasking.
I went one step further, in saying that the programs should reserve separate
arrays/variables for result data per connection, not per program, which makes
it even easier to make reentrancy functional, even with task threads.

But even though there is not much theoretical difficulty in implementation,
there would be a lot to do, both in kernel and in the TCP module.  So these
changes would take time to implement and then to debug.  At present Peter
has not had time to take part in this discussion, so nothing is decided.

Even if we do decide to adopt my ideas more or less as they stand, the work
on them can't commence at once.  First we must produce a new STinG release
that eliminates the problems discussed lately, so that servers run reliably.


----- snip ----- re: proper recognition of 'ownership'
>
>You're right of course, but the device driver guarantees just that. If you
>use its own open() function for opening the connection, the act_pd will be
>the one that tries to do so, because the kernel would have taken care for
>that (Andreas says so). This differs a lot from using p_run.

This only relates to calls made from an active thread of a user program.
Code running in TSR programs and interrupts are not properly recognized
by any multitasking kernel, which always considers them to be part of the
currently 'topped' application.  This means that they can have trouble in
accessing 'owned' resources that were 'opened' or 'created' at a  time
when some other APP was 'topped' than the one being so at time of access.

This problem is intrinsic to the usage of TSRs and interrupt driven code,
and also to the problem of APPs that share resources with such TSRs and
interrupts, or even with other APPs.  MagiC can't change that.

This means that if strict rules of ownership are applied, then TSRs and
interrupt routines are made powerless to do much good, and APPs are not
able to share resources as mentioned above.

NB: Just to avoid terminology problems, let me emphasize that by 'resources'
    above I do not refer to 'RSC stuff' at all, but to lower level resources
    like files, allocated RAM blocks, STinG connections, etc.
(This wasn't for you Vassilis. You knew that. But some others might not.)


>----- snip ----- re: The first new argument to STinG *_open calls
>
>Fine with me up to now, but I think Peter can tell us more on the internal
>connection structures. Did I get it right, that the API should be modified
>so that we will be able to 'pass' this extra pointer?

Yes, either through some new 'special case' of existing TCP_open and UCP_open
functions, or through two new functions added specifically for the purpose of
opening connections in unblocking mode.  Note that while I might not mention
UDP everywhere, it must be taken into consideration too, since CN* functions
are common to UDP and TCP, and clients need to use consistent programming.

UDP adaptions would be trivial however, since long term UDP connections just
do not exist below the API level.  Most responses are immediate and caused
by single packet transfers.  (Except for ICMP error responses)

Personally I favour new 'open' calls. The ones we have are already complex
enough without adding more special cases.  For the present I'll call them
TCP_open_d and UDP_open_d, where the '_d' part is an abbreviation of 'defer'
because the results of functions are deferred to the time when interrupt
driven code can produce final results.


>You imply that later I think with:
>"> because the 'opener' routine knows the variable
>">it used to open the connection, and will be looking there for the result."
>Is it what you mean ?

Correct.

The TCP_open_d function should store the ptr value into a new element of the
TCP connection struct, with NULL being used to indicate old mode, so NULL is
also what would be stored in this element by the old TCP_open function.

Thus all TCP functions that work on connections can always see whether they
need to use deferred mode or not, so no other API entry points need to be
modified.  Great modifications are needed of course, but only 'internally'.

The user APPs on the other hand should simply remember which result pointers
they used for each connection, since they do not have access to the structs,
and in fact they may need to read results after the interrupt driven parts
of TCP.STX have already released the connection struct RAM.  This is one
of the great strengths of this scheme, that it allows the TCP module to kill
off unwanted structs after storing results in a variable of the APP, so that
the APP can read the results even after the connection has ceased to exist.


>----- snip ----- re: the CON_result
>
>>Connections opened in the old ways should have the pointer to CONN_result
>>set to NULL by UDP_open or TCP_open, so that all API functions can know
>>that for these connections the old blocking methods must be used.
>>
>Sorry, can't follow. You don't mean of course that TCP_open doesn't belong
>to the API, so which are the other API functions ?

They are mainly CN*, TCP_send, TCP_wait_state, TCP_ack_wait, TCP_close.
Of course I was not speaking of the entire STinG API above, but only of the
API functions implemented for protocol modules, such as TCP and UDP.

What I meant was that they don't need any new arguments, because they already
have the connection handle as one argument.  Internally this is translated to
a pointer to the TCP connection structure which owns that handle, so in the
TCP module all API functions do have access to the CONN_result pointer stored
there by TCP_open_d or TCP_open.  If that pointer is NULL, then the functions
need to use the blocking 'old mode', but otherwise they should use the new
unblocking 'deferred mode'.

Other functions of the STinG API, that do not use connection handles, will
not be affected by these changes, and if they need to be made unblocking
(MANY of them already are completely unblocking), this will have to be done
separately.  I have already described one way I could do that for 'resolve'.


>To me this seems like the
>new call will be "TCP_open (rem_host, rem_port, bufsiz, new_ptr)" and the
>old, hmmm, like the old !

No, that is not quite right.  The old prototype looks like this:

int16  cdecl  TCP_open (uint32 rem_host, uint16 rem_port, uint16 tos,
			uint16 buffer_size);

The new prototype could look like this:

int16  cdecl  TCP_open_d (uint32 rem_host, uint16 rem_port, uint16 tos,
			  uint16 buffer_size, API_result *result_adress);

Here API_result would be a typedef defined by TRANSPRT.H as being:

typedef	union api_result
{	int16	res_s16;
	uint16	res_u16;
	int32	res_s32;
	uint32	res_u32;
	void	*res_p;
}	API_result;

This makes it easy to extract proper result data for each function type.


>So, (in case of old way) TCP_open will internally
>set this pointer to NULL. Is this what you mean ?

Yes, that's correct.


>----- snip ----- re: In unblocked mode... E_UNFINISHED etc
>
>Great so far. E_UNFINISHED is exactly the return value I'd like from certain
>API calls, just to be able to 'take a look' at them from the main APP loop.
>(without blockings and without threads).

Yes, in fact we already have some similar ability through CNget_info CIBs,
which I use in NetD to sense connection of servers, but that is quite limited
since most API functions still block the calling application.


>----- snip ----- re: explanation of the above
>
>I find it very good, but I have the feeling that it concerns Peter much more
>than the rest of us.

Of course it does.  At present only two real protocol modules exist, and he
wrote both of them, although I helped a bit debugging some parts of them.
He would have to do the major adaption work if my suggestions are accepted,
with all the help I can give him too of course.

But every client/server author is also concerned by this, since it affects
the API they'll have at their disposal for future STinG based projects, and
non-programming users will also be affected by the final results.


>From the description, I can't find any visible 'hole'.
>(and it's clear as already reading the pseudocode!).

I'm glad to hear it.
I've put a lot of effort into presenting these ideas in understandable form,
since that is naturally required if they are to gain acceptance.


----- snip ----- re: using suggested features from  multi-connective APPs
>
>For more simultaneously open connections (like Gary's POPSMTP server with
>up to 99 connections) I'd suggest the "More alternatives:" part in the
>original posting (or maybe Gary himself to tell us what he thinks about it).

Once we have an unblocking API mode I think the alternative choices will be
virtually unlimited.  Today all methods have to solve the blocking problems
by strange workarounds, which severely limits the options.


----- snip ----- re: own methods VS methods based on MagiC
>
>I just wanted to use available MagiC's feautures if that damned thing was
>found running in a machine (provided of course that general solutions could
>not be found).

I have no problem with that, but if we do that for MagiC then others will
want to have it done for MiNT too, and in both cases there must be different
ways for various release versions of those multitaskers.  Pretty soon we end
up in a morass of of special case adaptions demanding lots of debugging work.
That is the reason why Peter and I are so reluctant to do things that way.


>With today's proposal, I'm begining to believe that this is not anymore.
>It needs more examination, that's true, it needs to be tested, but I have
>the feeling that it *can* work.

It should, and even if we find some complications I think we can solve them.


>On the other hand, if a 'one for all' solution can also be the most
>efficient for each one of 'all', this can only be answered in the final
>testings or better by comparing *implemented* different solutions.

We obviously can't finish two separate versions just to dump the one that
proves less efficient than the other. Some major software companies could do
that sort of thing, placing dozens of system programmers on each project to
find out which is best.  We obviously do not have those resources.
So we have to decide which way to go without such knowledge...


>.... I'd say I agree to go to the direction of Ronald's proposal and trash the
>/dev/sting driver. At least for the moment.
>
>But I also thing that you have to move fast. STinG hasn't upgrated for ages
>(at least for people checking Peter's page) and many may believe that this
>is also a 'dead' project as many others. Needless to say that some of our
>software also desperately needs the changes. Thus time presses for a fast
>decision.

True, but my new API suggestions should not affect the upcoming release.
That needs to be completed before we even start on any unblocking changes.
We must release a version implementing the present API in such a way that
servers function completely reliably. This requires at least a fully fixed
TCP_ack_wait, and preferably also a fixed TCP_close.

Once such a public release has been made, that will be the time to seriously
consider the implementation of unblocking calls.


----- snip ----- re: butting my head agains the walls of limitations
>
>Absolutely ! If a fight can have such results, I'm begining to change my
>mind about regreting for started one ;-)

Naah, that wasn't really a fight, just a spirited argument...  ;-)
Such are often quite stimulating, as long as all involved manage to
control tempers well enough to prevent outbreaks of real fighting.
