Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.

History is littered with hundreds of conflicts over the future of a community, group, location or business that were "resolved" when one of the parties stepped ahead and destroyed what was there. With the original point of contention destroyed, the debates would fall to the wayside. Archive Team believes that by duplicated condemned data, the conversation and debate can continue, as well as the richness and insight gained by keeping the materials. Our projects have ranged in size from a single volunteer downloading the data to a small-but-critical site, to over 100 volunteers stepping forward to acquire terabytes of user-created data to save for future generations.

The main site for Archive Team is at archiveteam.org and contains up to the date information on various projects, manifestos, plans and walkthroughs.

This collection contains the output of many Archive Team projects, both ongoing and completed. Thanks to the generous providing of disk space by the Internet Archive, multi-terabyte datasets can be made available, as well as in use by the Wayback Machine, providing a path back to lost websites and work.

Our collection has grown to the point of having sub-collections for the type of data we acquire. If you are seeking to browse the contents of these collections, the Wayback Machine is the best first stop. Otherwise, you are free to dig into the stacks to see what you may find.

The Archive Team Panic Downloads are full pulldowns of currently extant websites, meant to serve as emergency backups for needed sites that are in danger of closing, or which will be missed dearly if suddenly lost due to hard drive crashes or server failures.

Collection: ArchiveBot: The Archive Team Crowdsourced Crawler

ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites).

To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel operator permissions in order to issue archiving jobs. The dashboard shows the sites being downloaded currently.

There is a dashboard running for the archivebot process at http://www.archivebot.com.

ArchiveBot's source code can be found at https://github.com/ArchiveTeam/ArchiveBot.

Defining "Free and Open Standard"

FoldUnfold Table of Contents Defining "Free and Open Standard" Politicization of terminology Vendor capture Free and Open Standard A canonical definition Measuring immunity The development process The implementation process The certification process The deployment process The authorization process Abstract Terminology

Politicization of terminology

What is an open standard? The Wikipedia page shows many definitions, which specify characteristics of a specification, or of the processes that produce it and make it available.

To understand why there is no single agreed definition, and to let us build a canonical definition, we can start with two observations:

1. The standardization process is driven by two conflicting economic motives. Established vendors see standards as a route to direct profits, while the market at large sees standards as a route to lower costs.
2. As the economic has become digital, governments - both as users and regulators - have become engaged in the conflict between these two interest groups.

The definitions collected on Wikipedia can be grouped into those made by vendors, and those made by the rest of the market. The variation in definition comes from the various viewpoints expressed (e.g. W3C focuses on process while Denmark focuses on user cost).

We, the Digital Standards Organization, explicitly take the side of "the market at large". We do not accept the definitions of "open standard" produced by vendor bodies, including W3C to some extent. We do not accept the attempts of some legacy vendors to stretch "open standard" to include RAND-licensed standards.

An open standard must be aimed at creating unrestricted competition between vendors and unrestricted choice for users. Any barrier - including RAND, FRAND, and variants - to vendor competition or user choice is incompatible with the needs of the market at large.

Vendor capture

Looking at the standards "wars" that plague the telecoms and media industries, we see that the fight is not over technical quality or process. The war is to capture markets through the use of standards backed by patent pools. The goal of vendors who engage in such standards processes is to become part of the cartel and to kill or capture the other cartels.

Vendor capture is thus a persistent and fundamental danger to a standardization process. Incompetence and human error are fixable by collective processes and time, as long as standards are free to improve. Vendor capture occurs at any stage in the life-cycle of a standard, from development through to long-term use:

• Vendors capture the standards development groups and exclude open participation in the development process. The standards which result become dumps of the vendor's software, which gives significant advantage to vendor.

• Vendors capture the standard specification through copyright and reduce competition from smaller competitors by making it expensive to simply acquire the text.

• Vendors capture the market by introducing patented technology into the standard, silently or explicitly. When many vendors do this, they create franchise standards ("reasonable and non-discriminatory" at the best of times) that enable cartel-like constructions.

• Vendors capture the market by silently extending open standards. Users who thought they were getting an open standard may instead be buying lock-in.

• Vendors capture the standards authorities. They do this by setting up their own (like ECMA) or lobbying existing ones (ISO) to promote and accept their advantageous standards.

• Vendors capture existing open standards. They do this by acquiring key patents, long after the market has committed to the standard. Often, such patents were held originally to "defend" the standard.

It is not enough for a standard to be open, it must also be resistant against all these forms of attack.

Free and Open Standard

The term "open" itself has many degrees of meaning from a high wall with a crack in the door to a field with no walls at all. We wish to secure the terminology by adding the word "free", in both its meanings:

• Freedom to use, improve upon, trust, and extend a standard over time.
• Freedom from all costs and tariffs associated with the above freedoms.

The ambition of freedom in this case is the removal of barriers, of friction, and of costs. It is also the ambition of remaining free over time, especially as the value of the market based on the standard increases.

A canonical definition

From the above analysis that the interests of established vendors diametrically oppose those of the market at large, we can make a simple canonical definition of "free and open standard" as follows:

• A standard is a published specification.
• It is a free and open standard if it is immune to vendor capture at all stages in its life-cycle.

This definition deliberately expresses the threat. A less direct definition ignores the threat and lists the primary properties of a free and open standard and its process:

• The standard is adopted and will be maintained by a not-for-profit organization, and its ongoing development occurs on the basis of an open decision-making procedure available to all interested parties.
• The standard has been published and the standard specification document is available freely. It must be permissible to all to copy, distribute, and use it freely.
• The patents possibly present on (parts of) the standard are made irrevocably available on a royalty-free basis.
• There are no constraints on the re-use of the standard.

Measuring immunity

Immunity from vendor capture, if this is indeed the defining characteristic of a free and open standard, can be measured. This is very useful because it gives us a tool to measure how "free and open" a standard is, and to offer concrete suggestions for improving any given standards process or specification.

We look at each aspect of the standardization life-cycle, from development to wide-spread use, ignoring quality issues, which we assume an open process will detect and improve.

The development process

In which experts agree to work together to standardize an area of technology:

• What is the cost of entry for experts? If not very low, independent experts are excluded.

• Do vendors form the majority of developers? If so, the standard will be rapidly captured.

• Do all developers grant their copyrights and patents to the standard by contract? If not, copyright and patent claims may be used later to capture the standard.

• Can others freely fork the standard under a share-alike license? If not, a captured standard cannot be freed.

• Is the standard published frequently during development? If not, external reviewers are excluded and vendors can more easily capture the standard.

• Is the process transparent and open? If so, vendor capture is easier to see and correct.

• Is there a process for accepting specification translations? If not, foreign vendors are put at a disadvantage.

• Is the development process hosted by a not-for-profit organization? If so, it is relatively safe from vendor capture.

While "forking a specification" may seem counter-productive, we consider this an essential freedom at this stage in the standard life-cycle. Sometimes, a standard will become "loaded" with features that favor one vendor and exclude others, and the best way forward is to fork the standard, strip-out those features, and create a competing standards process. The simple possibility of such competition forces a more honest process.

The implementation process

In which vendors - often also standards developers - take provisional specifications and use them to make products:

• Is the specification text clear and well-engineered? Vendors often inject complexity into specifications to create barriers to competing implementations.

• Are there reference implementations freely available? This promotes a level playing field. Copy-left implementations ensure that closed source derivatives that silently modify the standard are not possible.

• Is the standard free to implement for any purpose? If particular license conditions apply, this is easily used to exclude competitors.

• Can the specification be freely acquired and distributed? If not, cost can be used to create barriers for smaller competing vendors.

• Is there a process for contributing change proposals back into the standard? If not, it is easier for vendor-developers to capture the standard.

• Are there outstanding patent or copyright claims on the standard? If so, vendors are unable to freely implement the standard.

The certification process

In which the standards group, or other groups, help define what is a valid implementation and help the market measure and compare implementations:

• Are there conformance tests freely available? This promotes a level playing field among vendors. We would argue that copy-left licenses are most appropriate for conformance tests.

• Is certification done by a not-for-profit body? If not, it is trivially captured through acquisition.

• Are there clear rules for building subsets and extensions? If this is part of the standard specification itself, users can demand better behaviour from vendors.

• Are trademarks used to enforce implementations? If not, it is easier to capture users through extended-subset implementations.

• Are trademarks held by a not-for-profit? If so, they are more immune from capture.

The deployment process

In which users use standards-based products to construct their own business infrastructure:

• Is there a real market of competing vendors? If not, this is a sign that the standard has been captured.

• Is the standard mature and well-certified? If not, users can be captured by vendors who sell premature, incompatible implementations.

• Is the standard free from patent claims? If not, users are trapped by using the standard in any way at all.

The authorization process

In which an authority declares its support and recognition of the standard:

• Is the authority backed by the state, or by vendors? Vendor-backed authorities are by definition captive, and will readily promote franchise standards as "open".

• Is the authority meritocratic? That is, are its decisions taken by experts with proven track records in standards recognition? If not, the authority can be easily captured by vendors.

• Does the authority have clear, transparent rules on copyright and patent claims? If not, franchise standards can be injected, capturing users by deceit.

• Does the authority have a clear definition of "open standard" that represents the economic interests of the market at large? If not, vendors can more easily defeat open standards with franchise standards.

Edit | Attachments | Tags | Source | Print

Abstract

Many groups and individuals have provided definitions for "open standard" that reflect their economic interests in the standards process. We see that the fundamental conflict is between vendors who seek to capture markets and raise costs, and the market at large, which seeks freedom and lower costs.

We argue that standards lie on a spectrum ranging from franchise standards at one extreme to free and open standards at the other. Vendors work hard to turn open standards into franchise standards. They work to change the statutory language so they can cloak franchise standards in the sheep's clothing of "open standard".

A robust definition of "free and open standard" must thus take into account the direct economic conflict between vendors and the market at large. Such conflicts do not end when a standard is published, so a free and open standard must also be immune from attack long after it has been widely implemented.

We therefore define a free and open standard as "a published specification that is immune to vendor capture at all stages in its life-cycle", in short, and in long we define the fundamental properties of such a specification.

Of all threats to open standards, software patents remain the one that cannot be eliminated by careful design of process or contract. No agreement can prevent a third-party - "troll" - from asserting a patent claim against a successful open standard. Current patent case-law in Europe, Japan and the US heavily favours the interests of patent holders, over the interests of the market and society at large. Until this imbalance is rectified, all open standards are vulnerable to capture through patent ambush, and we expect to see this occur more often, as traditional routes to capture are closed off by better awareness and practice.

Terminology