
Date: Sat, 28 Apr 90 04:50:43 +0200
From: klemets@sics.se
Subject: NNTP for NOS
To: tcp-group@ucsd.edu

Because of this recent discussion about News support in NOS, I thought
it would be a nice idea to implement NNTP in NOS.
I have only done a client yet however.
The client is controlled by the following commands:

	nntp addserver <nntpserver> <interval>
  where <nntpserver> is the hostname of a server from which you want
  to fetch the news. <interval> is the interval between polls, 
  measured in seconds.

	nntp dropserver <nntpserver>
	nntp kick <nntpserver>
	nntp listserver
  What these commands do should be obvious.

	nntp groups <list of groups>
  where <list of groups> is a series of USENET newsgroups, separated
  by blanks or (preferably) commas. This command controls which
  newsgroups you will receive. By default you will receive all newsgroups.
  You should probably use this command to restrict the number of groups,
  unless you have unlimited disk space. The "*" and "!" signs are
  supported. For example
	nntp groups rec.ham-radio*,!rec.ham-radio.swap
  will make you receive all groups whose name matches rec.ham-radio
  except rec.ham-radio.swap.

The NNTP client asks the server for the Message-ID's of any new
articles that have arrived after a certain date. The client compares
the Message-ID's that it gets from the server with ID's of old articles.
These old ID's are kept in the /spool/news/history file. The client
then requests those articles that it does not already have.
The date and time when a server was last contacted is stored in the
file /spool/news/nntp.dat. The date can be changed manually if necessary.
Don't forget to create the /spool/news directory, by the way.

The articles are not stored as separate files, as is common in UNIX.
Instead, they are converted to mailbox format and stored in one
separate file for each newsgroup. The reason for this is that you are
then able to use the BBS already inherent in NOS to read the articles.
It is also possible to use BM for that matter.

A typical entry in the /spool/areas file would be:

rec/ham-radio/packet   The packet radio newsgroup

The client still needs some enhancement. In particular, it would be
nice if it could post articles too. I might have to use the stat()
call. Has it been fixed for DesqView yet?

The client should also make it possible to just grab the header of an
article, and let the user decide if the rest of the article should be
transfered. But this would require a special news reader utility.

The source for this is vailable with ftp from sics.se. The filename
is archive/packet/ka9q/nos/nntp.arc.

/Anders SM0RGV

----------------------------------------------------------------------------

The new changes (as of May 12, 1990) are described next, followed by older
changes:

Newsgroup name adjustment
    Previously, newsgroups whose names contained characters that were not
    valid in DOS filenames would not get created.  I now map any character
    that is nonalphanumeric and not '-' into an underscore ('_').

Locking of nntp.dat history files
    Previously, it was possible for a session to open while another was still
    in progress; the result would be that two processes would be trying to
    update these files.  I've added appropriate mlock() and rmlock() calls to
    prevent this.  If nntp.dat is locked when I try to first read it, I close
    the session and try again later.  If it's locked when I try to update it,
    I simply don't update it; the next session to that server will get the
    old date in the NEWNEWS command, but the old message ids will be found in
    the history file and so not transferred.  If the history file is locked,
    I abort the session; this is not a great solution, but is better than the
    alternative.

Updating of nntp.dat changed
    Instead of overwriting it in place (which for reasons unknown didn't seem
    to work for me), I now recreate it.

Enhanced NNTP DIR command
    You can now give a second argument, which becomes the Newsdir directory.

Changed date stamp on 'From ' lines
    used to be mm/dd/yy hh:mm:ss, now it's just elapsed seconds since origin
    of GMT.  Make it both easier to parse and easier for expire to work with.

Buffers in getarticle() now malloc'd
    used to be on stack -- Phil pointed out it's better to mallocw() them,
    so I do.

The following describes my changes to Anders Klemets' NNTP client for
the KA9Q NOS package.  The changes described herein are as of May 6, 1990.
                              --Bernie Roehl
                                broehl@sunee.waterloo.edu

NNTP TRACE command
    The syntax is the same as for SMTP TRACE.  There are four levels of
    detail that can be specified:
        0 - no tracing
        1 - only serious errors reported
        2 - same as 1, plus transient errors reported
        3 - same as 2, plus session progress reported
        4 - same as 3, plus actual received articles displayed
    Note that (1) is the default, and corresponds to the way things were
    in Klemets' original code.

More error messages
    I've added some error reporting (most of it subject to the nntp trace
    setting).

NNTP DIR command
    Allows you to set a spool directory other than the mail directory.
    Note that this is *not* the "Newsdir" directory (where history and
    nntp.dat are kept) but rather the spool directory where the articles
    themselves are kept.  The idea is that you could store articles on
    another drive.

Connection windows
    You can now specify a range of times at the end of an "nntp add" command,
    in the form "nntp add localserver 600 22:00-23:00", which would cause
    the nntp connections to be made only between 10 pm and 11 pm local
    time.  (NOTE: uses gettime() call, which is DOS-specific).

Site-specific newsgroups
    You can now override the "nntp groups" setting for specific servers.
    For example,
          nntp groups !rec.pets.gerbils
          nntp add myfeed 1800
          nntp add anotherfeed 7200 2:00-5:00 rec.pets.gerbils
    would take a full newsfeed from site "myfeed" every half-hour, but
    would not take rec.pets.gerbils.  The group rec.pets.gerbils is brought
    in from site "anotherfeed" every 2 hours, but only connecting between
    2 and 5 am local time.

Accumulating NNTP ADD commands
    Because additional information is now added on to NNTP, the commands
    can get quite long.  Therefore, "nntp add" commands now merge their
    values.  For example,
           nntp add myfeed 1800 rec.pets.gerbils,6:00-9:00,7:30-9:30
           nntp add myfeed 7200 comp.binaries.eniac
           nntp add myfeed 600
           nntp list
    will produce the following:
        myfeed        (595/600 7:30-9:30) rec.pets.gerbils,comp.binaries.eniac
    Note that the interval value *replaces* the previous one, as does the
    window setting; the groups accumulate (up to 512 bytes total).
    To reset the group list, "nntp drop" the entry and re-add it.

File Locking
    The mlock() and rmlock() functions within NOS are now used to lock the
    news files while they're being updated.  This is for use in conjunction
    with a (yet-to-be-written) news-expiry program.

Date/time stamp on 'From' lines
    'From' lines in the news files now have the date and time appended,
    in the form MM/DD/YY HH:MM:SS.  This for use with a (yet-to-be-written)
    news expiry program.  (Note: this is the *arrival* date of the article,
    not the date it was posted on!)

Increased stack space
    Even in the original nntpcli.c, the stack allocated was not really enough.
    I've bumped it to 3k (up from 1k) because of all the array[NNTPMAXLEN]'s
    in getarticle() and elsewhere.

Updating of nntp.dat only if the session is succesful
    Otherwise, we might lose articles.

That's about it.  Be sure to read Anders Klemets' original description,
in klemets.txt, and of course read through the code to see how it works :-)

If you have any questions about the code, send mail to me at
broehl@sunee.waterloo.edu, or to tcp-group@ucsd.edu.

I'm working on an expiry program (as time permits...), and am thinking about
how to implement posting (using IHAVE).  Will keep you posted.
