----------------------------Original message----------------------------
From:         Stevan Harnad <harnad@PRINCETON.EDU>
Subject:      More on Usenet

Here is more information about the Usenet world, posted with permission
from Gene Spafford. He has also corrected my posting. Arbitron
statistics cover 800-odd Usenet groups, but there are actually close to
2000 in all. Arbitron surveys only the more global groups. I continue
to suspect that Usenet rather than Listserv will turn out to be the way
of the future for scholarly electronic communication, especially
electronic journals -- and it's certainly the way libraries, eager
to cut costs and hasten the era of electronic journals, ought to
make the existing electronic journals available to its users now.
-- SH
------------------------------------------
Subject: Re: How to get arbitron data on sci.psychology.digest
Date: Fri, 06 Dec 91 11:48:31 EST
From: Gene Spafford <spaf@cs.purdue.edu>

You don't need Brian's permission, as it is info posted to a public
newsgroup.  However, you should cite it, of course.

Brian estimates that there are 40,000 sites on the Usenet.  I suspect
there are considerably more -- perhaps as many as 150,000.  Brian
estimates that there are approximately 1.8 million readers of at least
one Usenet group.  Again, I think this estimate may be low.  His
statistics show an average of 45 readers per site, which means most of
his statistics are gathered from large news machines; the number of
small systems vastly outnumbers the large systems, and thus I think
they are being underreported and undercounted.

The following is taken from one of Brian's postings:

                EXPLANATION OF THE MEASUREMENTS AND STATISTICS

Survey data is taken by having one person at each site run a program called
"arbitron", which looks at the news or notes files and determines the
newsgroups that the user has read within a recent interval. To "read" a
newsgroup means to have been presented with the opportunity to look at at
least one message in it. Going through a newsgroup with the "n" key counts
as reading it. For a news site, "user X reads group Y" means that user X's
.newsrc file has marked at least one unexpired message in Y. If there is no
traffic in a newsgroup for the measurement period, then the survey will show
that nobody reads the group. For a notes site, "user X reads group Y" means
that user X has been in the notesfile with the sequencer in the last 14 days.
The "14 days" interval for notesfiles corresponds to "unexpired" for news.

The "arbitron" program is periodically posted to comp.sources.d, or is
available from me (decwrl!reid). The notesfiles version of the program should
be available through standard notesfiles software distribution channels as
well.

SITES SURVEYED IN THIS SAMPLE

"This Sample" means the set of sites that have sent in an arbitron report
within the past "Readership measurement interval" days. In every case the
most recent report from each site is used. At the moment, some of the
readership reports are several months old. In future postings those reports
will have expired and will not be included.

The number in parentheses after the site name is the number of users that the
site reported. A value of (0) usually means that the software has been
configured to use the wrong technique for counting users at that site; a
report showing 0 users but 6 readers of rec.humor.funny is statistically
meaningful.

One might argue that the sample is self-selected, and thereby be biased. It
does in fact have a certain self-selection factor in it, because we only get
data from sites at which someone participates in the survey. However, we do
not require the participation of every user at a site, only one user. The
survey program returns data for every user on the system on which it was run.
Since there are an average of 30 people per site reading news, there is a
certain amount of randomness introduced that way. Of course, the sample is
biased in favor of large sites (they are more likely to have a user willing
to run the survey program) and software-development-oriented sites (more
likely to have a user *able* to run the survey program).

NETWORK SIZE

I determine the network size by looking at the set of sites that are
mentioned in the Path lines of news articles arriving at decwrl. This number
is consistently higher than the number of sites that posted a message (as
measured and posted from uunet) because it includes passive sites that are
on the paths between posting sites and decwrl. Each month I store the names
of the hosts that are named that month, and for this report I used the past
12 months worth of data.

There are 39319 different sites in the Path lines of articles that
arrived at decwrl in the last 12 months. There are 12528
different sites in the comp.mail.maps data, but comp.mail.maps tends to
include only one or two machines for each organization, leaving the rest
unmentioned. Also a large number of sites participate in USENET without
participating in UUCP.

I believe that 40000 is the best estimate for the size of USENET.
Because it is actually a measurement of the number of sites that have posted
a message or that are on the path to a site that has posted a message, it
will be slightly smaller than the number of sites that actually read netnews.
Any site that believes it is not being counted can just ensure that it posts
at least one message a year, so that it will be counted.

NUMBER OF USERS

The number of users at each site is determined in a site-specific fashion.
Sometimes it is done by counting the number of user accounts that have
shells and login directories. Sometimes it is done by counting the number of
people who have logged in to the machine in some interval. Sometimes other
techniques are used. This number is probably not very accurate--certainly
not more accurate than to within a factor of two.

ESTIMATED TOTAL NUMBER OF PEOPLE WHO READ THIS GROUP, WORLDWIDE

There are two sources of error in this number. The number is computed by
multiplying the number of people in the sample who actually read the group by
the ratio of estimated network size to sample size. The estimated total can
therefore be biased by errors in the network size estimate (see above) and
also by errors in the determination of whether or not someone reads a group.
Assuming that "reading a group" is roughly the same as "thumbing through a
magazine", in that you don't necessarily have to read anything, but you have
to browse through it and see what is there, then the measurement error will
come primarily from inability to locate .newsrc files, which can either be
protected or moved out of root directories. There is no way of measuring the
effect on the measurements from unlocated .newsrc files, but it is not likely
to be more than a few percent of the total news readers.

PROPAGATION: HOW MANY SITES RECEIVE THIS GROUP AT ALL

This number is the percent of the sites that are even receiving this
newsgroup. The information necessary to compute propagation was not generated
by early versions of the arbitron program, so the "basis" (number of sites)
used to generate the Propagation figure is smaller than the "Sites in this
sample" figure. A site's data will be used to compute propagation if either
(a) it reports zero readers for at least one group, or (b) it is using an
arbitron with an explicit version number that is high enough.

MESSAGES PER MONTH AND KILOBYTES PER MONTH

Traffic is measured at decwrl, in Palo Alto, California. Any message that has
arrived at decwrl within the last "Traffic measurement interval" days is
counted, regardless of when it was posted. Monthly rates are computed by
taking the total traffic, dividing by the number of days in the traffic
measurement interval, and multiplying by 30. Decwrl runs 2.10.3 news, which
does not store the "Date-Received", "Relay-version" or "Posting-version"
header lines; the amount of space occupied at your site might be higher, and
the number of bytes transmitted between machines is probably higher. By
definition this number is correct, because it is an exact measurement, but it
may differ from the traffic at your site by as much as 15% due to timing
differences and news version differences. Timing differences will be random,
but will average out in the long run. News version differences will cause a
systematic error that is additively uniform across all newsgroups, and which
therefore does not significantly affect ratios.

If a message is crossposted to several groups simultaneously, it is charged
only to the first-named group in the list. Note that this differs from the
statistics posted from uunet every 2 weeks: the uunet data charge a message
equally to every group that it is crossposted to.

CROSSPOSTING PERCENTAGE: WHAT FRACTION OF THE ARTICLES ARE CROSSPOSTED

"Crossposting" means to post the same article simultaneously in more than one
newsgroup. In genuine "news" systems crossposting is implemented with Unix
links and does not increase the storage or transmisison cost, though in some
other systems crossposted articles are unbundled and must be stored and
transmitted separately.

The "crossposting percentage" is the percentage of the articles in this group
that are crossposted to at least one other group. If every article in this
group is crossposted, the percentage will be 100%; if none is crossposted,
then the percentage will be 0%. The crossposting percentage figure does not
take the size of the article into account, only the number of articles.
Crossposting a 50,000-byte article or a 50-byte article both cause the same
tally.

COST RATIO: DOLLARS PER MONTH PER READER

The most controversial field in the survey report is the "$US per month per
reader". It is the estimated number of dollars that are being spent on behalf
of each reader, worldwide, on telephone and computer costs to transmit this
newsgroup. The rate of $.0025 per kilobyte is the same value used in the
UUNET statistics reported biweekly. It is based on discussions among system
administrators about the true cost of news transmission.

The cost ratio is computed as follows:

$US/month/reader = ($USPerMonthPerSite * numberOfSites) / numberOfReaders
$USPerMonthPersite = KBytesTrafficPerMonth * $USPerKByte * Propagation factor
$USPerKByte = 0.0025

Combining all these gives

$USPerMonthPersite =
    KBytesTrafficPerMonth * 0.0025
  = KBytesTrafficPerMonth / 400

Therefore:

$US/month/reader =
    (KBytesTrafficPerMonth * numberOfSites) / (400 * numberOfReaders)

The accuracy of this number is in fact better than the accuracy of the
participation ratio, because the source of error--the network size
estimate--is present both in the numerator and the denominator, and therefore
cancels out. The primary source of bias in this number comes from the bias in

the "estimated number of readers, worldwide", which is described above. Treat
this value as being accurate to within about 25%.

SITE PARTICIPATION

I would like to receive data from every site on USENET. The arbitron programs
(posted comp.sources.d along with this report) work on news 2.9, 2.10.[1-3],
2.11, and on many versions of notesfiles.

Brian Reid
DEC Western Research Laboratory, Palo Alto CA
reid@decwrl.DEC.COM
{uunet,ucbvax,sun,pyramid,att}!decwrl!reid