here is another of my 'infamous' verbose mails...  ;-)
But please try to endure, as I think the subject matter is worth it.


on Thu 16-03-2000 10:17 Dan Ackerman wrote:
>On Thu, 16 Mar 2000, Ronald Andersson wrote:
>
----- snip ----- re: starting work on Doom, Heretic, etc.
>
>I had thought about this myself.  I just don't have the time.  I'm
>already working on a few other group projects and don't want to overcommit
>myself too much.  I already invision very litle sleep in a couple of
>months as some of them heat up.

It is a common problem, and I do have some other commitments myself, but that
is also why I wrote 'not too soon', as I might have to shelf this stuff for a
while to deal with other things with higher priority than games.


>But if you have any questions in regards to BSD sockets please
>don't hesitate to ask.  I've gotten fairly fluent at reading it and
>translating to our API.

The only really relevant question here is whether or not *any* comprehensible
docs exist for this stuff *anywhere*.

The MintNet docs do not even contain any list of the available functions with
their prototypes.  Anyone interested in this will have to (try to) puzzle it
out from various source code files or split 'man' pages, which often refer
to some crucial subpage not included.  This is chaotic !

The same description also applies to the Linux installations I have tried,
so the problem is not tied to MiNT exclusively, but appears to be a common
characteristic of 'Unices' (or whatever the plural should be ;-).


>The one thing we need to agree on is an implementation of
>select().  I've looked at my end and it would not be very hard to do,
>simply an isertion in a few locations and a routine for the notification.

I'm not sure if it would be any use trying to make a 'select' implementation
that directly copies what is done in a socket model.  That is often used in
conjuction with 'fork' statements that split the execution into several
threads, so one can do general APP management while other(s) attend to the
networking.  Such methods are not well suited to STiK and STinG anyway, as
they must also work under SingleTOS, where real forking is not practical.

But we can implement something that can serve a similar purpose, though it
still requires rewrite of any parts that rely on forking.

For many cases though, 'select' is simply used to wait for incoming data of
a few connections, and then one may as well recode it to use CNbyte_count a
few times instead.


>This is the one area where we currently really need to mangle BSD
>code to have it work.

Ok, and the main problem with it, as I see it, is that it assumes that the
socket descriptors can always be completely identified by 32 bit bitmaps.
This in turn implies that the descriptors are local to each application,
which is not the case for normal STiK/STinG connection handles.

What we need here is a way for any application to get a local handle for each
connection, which can then be kept within the range 0..31.  Given such local
handles it should be fairly easy to code a 'select' function similar to that
in the 'Mintlibs', but using STiK compatible interface functions.
(An example of how I'd do this follows further below.)

One crucial decision here is whether this implementation really needs any
changes at all in the STiK/STinG kernels.  I think it does not.

Remember that the 'select' of Mintnet is not a part of the kernel at all
(though Fselect is), and it is not even part of the 'sockdev' code. Instead
it is just one more function of the 'Mintlibs', linked in with each APP that
is to use this function.  We can easily do something similar, simply by
defining a '.c' or '.h' inclusion file (as we use no special lib).
That is the implementation method which I prefer for this function.


Regardless of whether it is to be implemented via kernel or simply via
inclusion files, it is obvious that we also need to  add some new function(s)
(implemented the same way as 'select'), whereby the bond can be created
between 'local' connection descriptors and the 'global' ones used by kernel.
These 'bonds' can then be registered simply by storing the 'global' handles
in a 32 word array, at index positions corresponding to the 'local' handles.

Note however, that with a kernel implementation it must maintain the arrays
for any number of APPs, and there must also be a function to release them.
That is a major reason why I prefer the method of implementing this stuff
outside of the kernels, simply by providing source inclusion files.


Once we do have a system for associating global handles with local ones,
we also have to deal with the fact that the main APP code will only use
the local ones to refer to any connection. This means that we must provide
some simple means to use those from any STiK/STinG function.  But I don't
think we need any new functions or 'wrappers' for that, as it can be done
simply by using an array reference (to the array mentioned above).  This
is yet another thing that is simplified by implementation through inclusion
files rather than through kernel, as the array storage is local to the APP,
and can easily be referred to directly.  Not even a pointer is needed.


So my proposal is to deal with this entirely by providing an inclusion file
that defines three things:

1:  An array of 32 words that will be used to associate between the local
    connection handles used by the APP and by a new 'select' function.

2:  A new function, to be used after each xxx_open call to create the
    local handles, by storing the global handle in the first free entry
    in the array. The index of that entry is returned as the local handle.

3:  The new 'select' function, modeled after the one in the 'Mintlibs',
    but using STiK interface calls, and accessing the array mentioned
    above to solve the bitmap problem.


Alternatively, wrapper macro calls may be used to implement pseudo functions
(eg: local_TCP_open and local_UDP_open) which makes use of the extra function
in '3:' automatic.  But in porting any socket oriented program we still have
to patch the parts that create the socket etc, so it doesn't really matter
how we solve this, as we still can't use exactly what the original source
contains.  I myself prefer to use a separate function, so that is how it is
done in the example below.


NB: The example pinpoints one lack of the STiK/STinG client interface, which
    I have not considered before, but which may be very important to some
    game programs, to avoid wasting time on TCP_send calls that are doomed
    to fail when the buffer is full.  We need a good way to test for that
    condition, and while we're about it we really should allow the real
    space available in that buffer to be returned.  This will be the value
    (Buffer_size - size_of_unACKed_data) for TCP, and for UDP it will always
    be 512 except when the connection is in passive state, awaiting the first
    incoming packets.  At that time the value should of course be zero, just
    as it should for TCP_connections in TLISTEN state.

    This could be done by a TCP_info/UDP_info extension, though I am not sure
    how this suits you and STiK.  It would be best to do it in a compatible
    manner, so please give me your views on this.


Here is an implementation example:  (I'm in a coding mood now ;-)
/*----------------------------------------------------------------------------*/
/* Start of inclusion file                                                    */
/*----------------------------------------------------------------------------*/
int16   local_cn[32];               /* This is the association array */
int16   cn_init_flg = 0;            /* 1 flags that the array is valid */         
int16   STiK_error = 0;             /* This is for STiK error codes */

typedef uint32  fd_set;             /* Here are two Mintnet-like typedefs */
typedef struct  timeval
{   long    tv_sec;     /* seconds */
    long    tv_usec;    /* microseconds */
}   TIMEVAL;
/*----------------------------------------------------------------------------*/
/* Start of function:   set_local_open                                        */
/*----------------------------------------------------------------------------*/
/* This function is the one to use directly after each TCP_open or UDP_open,  */
/* to ensure that the connection is available for 'select' (further below).   */
/*----------------------------------------------------------------------------*/
int16   set_local_open(int16 global_conn)
{   int16   i;

    if  (init_flg)
    {   for (i = 0; i < 32 && local_cn[i]; i++)
        ;   /* NB: This is an 'empty' loop.  We just want the index 'i'. */
        if  (i = 32)
            return -1;  /* Only 32 connections can be handled */
    }
    else
    {   for (i = 0; i < 32; i++)
            local_cn[i] = 0;    /* this initializes the array */
        i = 0;                  /* The new connection will be local_cn[0] */
    }
    local_cn[i] = global_conn;  /* And here the association is complete */
    return i;
}
/*----------------------------------------------------------------------------*/
/* End of function:     set_local_open                                        */
/*----------------------------------------------------------------------------*/
/* The macro below is the complement of the function above. It should be used */
/* when a connection is no longer wanted for select, mostly after closing it. */
/* Note that the 'x' argument should be the 'local' handle, not the 'global'. */
/*----------------------------------------------------------------------------*/
#define clr_local_open(x) local_cn[x] = 0;
/*----------------------------------------------------------------------------*/
/* Start of function:   select                                                */
/*----------------------------------------------------------------------------*/
int16
select(junk, rfds, wfds, xfds, timeout)
    int16   junk;
    fd_set  *rfds, *wfds, *xfds;
    TIMEVAL *timeout;
{
    uint32  start_time, time_out;
    int16   i, test, retval = 0;
    uint32  rsel,wsel;              /* for original request bits */
    uint32  rshf,wshf;              /* used for bit tests by shifting */
    uint32  rflg = 0, wflg = 0;     /* used for result bits */

/* NB: I only cater for 'rfds' and 'wfds' here as I honestly don't know    */
/*     what (if anything) 'xfds' would mean in networking (execute what?)  */
/*     The Mintlibs don't explain 'junk' at all, so I simply ignore it too */

    start_time = TIMER_now();       /* Sorry, I can't do this for STiK */

    if  (timeout)
    {   time_out = timeout->tv_sec*1000 + timeout->tv_usec/1000;
        if  (time_out == 0L)
            time_out = 1L;      /* change 0 to 1 for unblocking poll */
    }
    else
        time_out = 0L;          /* Use 0 for eternal wait (like Mintlibs) */

    rsel = (rfds) ? *rfds : 0L;
    wsel = (wfds) ? *wfds : 0L;

/* The loops below are a bit complex, but that is needed to make them fast */
    do
    {	for (   rshf = rsel, wshf = wsel, i = 0;
    	        (rshf | wshf);
    	        i++, rshf >>= 1, wshf >>= 1
    	    )
        {   if  (rshf & 1L)
            {   if  ( (test = CNbyte_count(local_cn[i])) != 0)
                {   if  (test < 0)
                    {   if  (test != E_LISTEN)
                        {   STiK_error = test;
                            return -1;      /* APP might not like STiK codes */
                        }
                    }
                    else
                    {   retval++;           /* increment success count */
                        rflg |= 1L << i;    /* set one success bit */
                    }
                }   
            }
            if  (wshf & 1L)
            {   /* Ooops! STiK has no defined way of testing send readiness */
                /* Neither has STinG, since they are still closely related. */
                /* A simple workaround is to consider it ready at all times */
                /* but writing may still fail if 'buffer' is full...        */
                /* Also, we must forbid it in listening state, so we must   */
                /* Still use CNbyte_count here too                          */

                if  (test = CNbyte_count(local_cn[i])) < 0)
                {   if  (test != E_LISTEN)
                    {   STiK_error = test;
                        return -1;      /* APP might not like STiK codes */
                    }
                }
                else
                {   retval++;           /* increment success count */
                    wflg |= 1L << i;    /* set one success bit */
                }
            }
        }   /* Here ends the 'for' loop */

/* This is a good place to insert 'time_wasters' like Syield or appl_yield,  */
/* to share time with other tasks/threads (if available), but their legality */
/* is very 'iffy' under some conditions, so I won't define how to do it now. */
/* If such things are inserted here, then some of the 'while' conditions are */
/* to be moved up (break) so time is not wasted before successful returns.   */

    } while (   (retval == 0)
            &&  ((time_out == 0L)  ||  (TIMER_elapsed(start_time) < time_out));
            )
    /* Sorry, I know STiK needs other code for timing, but this is the best */
    /* way to do it for STinG, as it will work under all conditions.  Other */
    /* ways to do it usually depend on CPU super/user state and other stuff */
    /* Please describe a suitable method with STiK, so I can use that in my */
    /* client code when STiK is detected (I will use the above with STinG). */

    if  (rfds)          /* if read tests were requested */
        *rfds = rflg;   /* set result flags for reading connections */
    if  (wfds)          /* if write tests were requested */
        *wfds = wflg;   /* set result flags for writing connections */

    return retval;      /* return count of successful tests */
}
/*----------------------------------------------------------------------------*/
/* End of function:     select                                                */
/*----------------------------------------------------------------------------*/
/* End of inclusion file                                                      */
/*----------------------------------------------------------------------------*/

-- 
-------------------------------------------------------------------------
Regards:  Ronald Andersson                  mailto:dlanor@ettnet.se
http://dlanor.atari.org/    ICQ:38857203    http://www.ettnet.se/~dlanor/
-------------------------------------------------------------------------
