To: stik@on-luebeck.de
Subject: Re: [2] STIK: New proposal: open development of Stik 2.x                            


On Fri, 17 Jan 1997 02:23:32 Martin Mitchell <mem@mode wrote: 
>
>On Thu, 16 Jan 1997, Ronald Andersson wrote:
>>
>> Neither of the module loading methods can be considered a hack. And why
>> should they ?  Both methods are used by various parts of TOS as well as
>> by other operating systems (incl MagiC).  You might as well argue that
>> TOS patch TSR's shouldn't be loaded by Pexec, but have a special format.
>
>Neither is a hack, however the Pexec() system is lazy, inefficient, and
>flawed.

'Lazy' :
    This is partially true, since using it means not having to invent,
    or to debug, some other method of loading.  Some call this 'efficient'.

"Inefficient" :
    This is only true in regard to module file size.  Run-time speed and
    size can be optimized regardless of loading method.

"Flawed" :
    This is patently untrue.  There is nothing inherently flawed in loading
    modules with Pexec.


----- snip ----- re: arguments favoring the Pexec method
>> 
>> There are some more, like the fact that a program started by Pexec has its
>> own basepage.  This is not only a RAM cost, like you describe it to be, but
>> also gives some protection from certain types of crashes.  It also allows
>> the module to own RAM and GEMDOS channels separate from the loading program.
>> The value of these abilities may be debatable, but they are nevertheless
>> abilities, and will be lost if Pexec is not used.
>
>These 'abilities' are worthless really. Just look at the so-called 'crash
>protection' you refer to, by having a separate basepage. The way STX
>modules work, by linking function pointers into STiK's structures, if an
>STX module crashes, these pointers will be rendered invalid, and so STiK
>will crash. Or to put it another way, if for some reason an STX module is
>terminated abnormally, then STiK will crash when it next tries to use any
>functions from that module.

That is wrong.  When a program called by Pexec bombs, the calling program
gets an error code which tells it that something went wrong.  The proper
thing to do then is of course to unhook all vectors affected by it. If a
similar error occurs with your scheme the STiK program itself will be the
one terminated, and although it may be possible to 'clean up' after it by
removing the XBRA vectors etc in a routine linked to etv_term, it will not
be possible to make a full recovery and keep STiK in memory.  That ability
is only available if the module has a basepage separate from STiK, which is
one of the reasons why I call that basepage an 'investment'.

NB: I am not saying that the early beta's you've already seen, or even the
    ones you are going to see for quite a while yet, have such protective
    code.  This will be added once the new STiK is functional, together
    with the speed optimization techniques I have repeatedly described.
    Trying to do all that on this early stage would only delay development
    of the actual communication protocol routines.


>Having separate memory allocation is also of little use. If we are
>assuming standard TOS to be a minimum operating standard for STiK, then
>the GEMDOS memory management is so terrible, that it is not worth
>releasing little blocks of memory. MagiC does much better, but I think we
>all agree that STiK should run on standard TOS.

This is also wrong, and in three ways.  The bugged and inefficient management
routines you speak of were debugged in TOS 1.4 (and all later), which means
that the large majority of users do not have them.  Also, the fact that some
users will not benefit from an improvement does not make the improvement
useless for the rest of the users, even if they were a minority.  Finally
(and more relevant to this discussion) the release of RAM via Ptermres, which
I suggest be used to release init routines and data of ASM modules, involves
only the shrinking of a RAM block to a smaller size, which has always worked
quite efficiently in all TOS versions that I know of.


>As for the 'program flags' then they may easily be emulated in a custom
>module system with Mxalloc().

True, although the code will need to test whether that function exists at all,
since it does not do so on the majority of systems.


----- snips ----- re: arguments against the Pexec method

>There are yet more arguments against the Pexec() method. Please read on.

Oh, I will, but those mentioned so far are either incorrect, irrelevant, or
of very low importance, so my views on this are as yet unaltered.


>Another problem with the Pexec() method, is that it is not valid using
>memory managers with MMU protection. Basically, it involves one kind of
>process (STX modules) overwriting data in another process (the STiK
>kernel) so they can interoperate. This kind of activity is forbidden in
>such systems! We need the function pointers installed in the STiK
>structures for speed reasons, but they should be installed by parts of the
>STiK process, as a custom module system would do, not by any other program
>that could come and crash STiK just by trashing the information available
>through the value of the STiK cookie, such as an errant STX module.

You seem to be under the impression that your scheme would allow the use of
'readable' protection mode as opposed to 'global' for the Pexec method.
In this you are quite wrong.  The 'global' mode is the only one functional
for any interrupt based system, which is what STiK has to be regardless of
the module loading method.  Therefore the status of STiK-owned RAM _MUST_
be global, with either scheme.


----- snip ----- re: flexibility and efficiency
>> 
>> All features possible to implement with a custom system can also be equally
>> well implemented using the Pexec method.  Why call the former more flexible ?
>
>No, they cannot be equally well implemented. RAM is wasted by the Pexec()
>method. I call the former more flexible because this waste can be
>eliminated.

That is a strange choice of words then, since I would rather call that storage
efficiency.  Flexibility, as I see it, is something entirely different.
Also, I do not agree with you that the RAM waste is unavoidable with the Pexec
method.  I know several ways of avoiding most of it, and so must others too.


>> Efficiency is one word with many meanings.  There need not be any runtime
>> (post-init) difference in speed efficiency between the two methods, and the
>> extra RAM cost you often have mentioned is not only a loss, but should be
>> regarded as an investment to gain certain abilities.
>
>There is a difference in speed, a custom module system will initialise
>faster than any system using Pexec(). Initialisation time is important to
>me, as well as runtime speed.

Do you really have time to get bored in a few milliseconds...?  ;-)


>The 'investment to gain certain abilities' is a waste of RAM, as I have
>shown above why these abilities are worthless in the context of STiK.

No, you have not.


>> It may not be very important, but I see no reason to exclude those who do
>> not use C or ASM from participating.
>
>I'm not at all surprised you agree. However, I repeat, if someone is
>capable of programming a STX module, they would also be capable of
>producing an executable in a custom format.

The custom format you speak of is not a standard Atari executable format, and
some languages insist on producing such formats only.  Modifying this for some
compilers is not necessarily so easy as you imply.


----- snip ----- re: Extra Pexec method RAM overhead costs

>By 'stub' I mean an STX module that installs itself, but provides no
>features. (ie the useful functions are empty). 2048 bytes for every module
>to load an install itself, is unacceptable overhead, and wasting RAM.

I fail to see the relevance of your figures.  Where and how does a simple
STX using Pexec method incur a cost of 2048 bytes more than by your scheme...?


>> >excluding the basepage size which must be added at runtime.
>> 
>> 'Is automatically added by Pexec' you mean, which I still regard as sort
>> of an investment, half of which is instantly refundable, with the other
>> half being the necessary key to using some TOS features in certain ways.
>
>There is no investment with the basepage either. Not many modules will
>need to use a DTA anyway, if they do, they may allocate their own.

What has that got to do with anything...?
The fact that Pexec activates a default DTA at basepage+128 does not mean that
this is the only possible use for it.  That 128 byte area is free for _ANY_
use that the program wants to make of it.


>> >The C startup code is present, scanning for commandline arguments,
>> 
>> That is up to the programmer, and is easily avoided in both C and ASM.
>
>I accept that, but if the standard C startup code is not used, then some
>custom commandline parsing code must be used, as commandline arguments are
>passed to STX modules. This uses space in each module, and is a further
>waste of RAM.

Not really, since commands passed to an STX imply that this STX must have
variable operational modes, that must somehow be initialized regardless of
the module loading method.  As always you seem to assume that your own scheme
can somehow eliminate initialization needs completely, which is wrong.
Before you abandon this assumption fair comparisons are impossible.

Naturally some STXs will only have one operational mode, implicit in the fact
that it has been called at all, and such STXs will of course not have any code
to analyze a commandline at all.  The same goes for all STXs whose variable
modes are to be selected at a later time (eg: by the CPXs or other programs).


>> >and the RAM occupied by this code is lost from the whole system.
>> >Executed once when the module is loaded, this is a clear waste of memory.
>> 
>> No, it need not be (see above), and in ASM it is rather simple to make
>> sure that all RAM containing init routines is released to the system when
>> the TSR init is completed.
>
>I reiterate, it is pointless to release little bits and pieces of RAM to
>the system in standard TOS, as the memory fragments through normal use
>already far too quickly.

No, in the case I describe it does not fragment at all.  The RAM block which
contains the STX program simply shrinks a bit, while the free RAM block that
follows after it grows by the same amount.  Also, we are not talking about
any 'little bits and pieces' here, but about _ALL_ code and data used only
for the initialization of the STX.  For some STXs this may well be a large
chunk of memory (regardless of module loading method).


>> >There will always be a need for such code to install a module,
>> >but its size should be minimized.
>> 
>> Sure, which can be done by a great variety of ways with the Pexec method
>> as well.  For example you can release all RAM holding init code and data
>> at the end of the program area, simply by specifying a smaller resident
>> size in the Ptermres call.  Note that this ability is lost without Pexec,
>> and is not normally possible from high level languages (requires ASM).
>
>Again, my point is that it is of little use to release small blocks of RAM
>in standard TOS.

In this you are wrong, two ways:

1: In some cases this will be a large area.
2: In this case it is not a separate memory block at all.


>It would be better to centralise the initialisation code
>within the STiK kernel, so that there is just one copy of it.

For some cases this would represent a savings, though it would also mean
limiting flexibility.   Some STXs may need exceptional initialization.
(eg: Setup extended hardware for aethernet etc.)


>> That is ridiculous.  If a module does not need those functions, their
>> code will not have to be present, regardless of module loading method.
>
>You're right, it is ridiculous! In the 'stub' STX file I have seen, at
>least the calls strcmp(), puts() and Supexec() were needed for module
>installation. Hence EVERY module using the Pexec() method needs these
>functions/bindings, leading to more RAM being wasted, either within the
>modules themselves, or as small blocks released to GEMDOS, which increase
>memory fragmentation.

How can you possibly judge _ALL_ modules for the Pexec method, and thereby
the method itself, by a single example of how an STX _can_ (not must) be
constructed...?  This is quite inappropriate.

Right now the top priority is to make the new STiK reach the functionality
which will allow its use in place of the current one.  Once this has been
achieved there will be time to produce the optimized versions.  Otherwise
we simply waste a lot of time by optimizing code (over and over) which is
replaced by the debugging needs.  That would make the development too slow.


>> Since each of the arguments you brought up above either have inadequacies,
>> or were based on erroneous assumptions, I must still disagree.  I still
>> feel that both methods are viable, with the same two differences that I
>> have mentioned previously.
>> 
>> 1: The custom load method will produce slightly smaller modules.
>> 2: The Pexec method allows some things that others do not.
>
>I agree with point 1, and agree also with point 2, but I will point out,
>that the things that Pexec() provides, are of little use in this context.
>(specifically, separate memory allocation and 'crash-protection' which I
>have shown to be worthless in the context of STiK)

No, you have not.


>I don't believe my arguments have inadequacies, or are based on erroneous
>assumptions. If you need clarification about any of the above points, I
>shall be happy to provide it.
>
>Please look at the facts and reconsider this issue. Particularly note the
>points I raised about the Pexec() method not being friendly for MMU
>protected program address spaces (important for good multitasking systems)
>and the points about memory allocation and what happens if a STX module
>crashes.

I have, and have commented each of these issues above.  I still do not see
the Pexec method as flawed or the custom loading as superior.


>> Nevertheless, I wish to remind you that many many projects have been
>> successfully implemented using Pexec loaded modules.  And as far as
>> I am concerned this has not diminished their quality one bit, though
>> I realize that most of them do not use the optimization techniques
>> that I would recommend for STX modules.
>
>Again, you must look at this in context. I think the Pexec() system is
>fine for some other applications, but not for STiK, where a high degree of
>optimization is needed. I have no problem with the Pexec() system in
>general, it is just that I believe it is the wrong system to use here.

That is precisely where I disagree with you.  I see no problem at all with
optimizing Pexec loaded programs.  That is after all what all normal APPs
and TSRs are, and no one ever said it was impossible to optimize them.


>> If you really do abandon STiK support over this issue, then I must, with
>> regret, conclude that you were not really serious about it to begin with.
>
>Well, I bothered to write this long reply in an attempt to ask you to
>reconsider your opinion, didn't I? I'll let you draw your own conclusions.
>
Actually I won't.  While you remain I will assume your intention to do so
continuously.  What you actually decide to do is your own business.


>> >This would encompass the following points:
>> >
>> >	- Snapshot release of STiK source code at least every 2 weeks
>> >	  during rapid development, every month otherwise.
>> 
>> Are you talking of public releases or beta releases ?
>
>I'm talking of neither public nor beta releases. I mentioned a snapshot
>release of STiK source code. This means a snapshot of the current STiK
>source. It may not work, it may not compile, and binaries should not be
>distributed with it, to discourage people testing unfinished code. It is
>primarily to allow some kind of code review process to be established,
>where programmers may look to check for bugs in the implementation. It's
>always quicker to find them earlier in the coding than debugging a program
>and finding them later.
>
The public release of source code is something entirely up to each individual
author.  Neither you nor I can make that decision, except for such code that
we ourselves have produced.  Since you mentioned the organized collection of
STXs in this connection you seem to imply that all STX authors should be
required to release all source.  I find it doubtful that all will agree.
I would agree myself, but the choice is necessarily up to each author.


----- snip ----- re: keeping the current API discussion policy

----- snip ----- re: summary of suggested future list policy

>The main difference is the snapshot release of STiK. As I mentioned
>above, this should not even contain a binary version of STiK. I feel it
>would aid in the rapid development of _more bug-free_ STiK code, which I'm
>sure you'll agree, is what we all want!

Yes, that is what we all want, but I am not certain if all authors will
agree to this method.  Most of the STiK package is SHAREWARE, and some of
the authors may not want to let their source code become semi-public in
this manner.

-------------------------------------------------------------------------
Regards:  Ronald Andersson                     mailto:dlanor@oden.se
                                               http://www.oden.se/~dlanor
-------------------------------------------------------------------------
