ABC.net.au

COLLECTED BY

Organization: Internet Archive

The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

Collection: Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What�s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa�s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available �warts and all� for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you�re hoping to do with it. We may not be able to say �yes� to all requests, since we�re just figuring out whether this is a good idea, but everyone will be considered.

TIMESTAMPS

The Wayback Machine - https://web.archive.org/web/20110629024412/http://www.abc.net.au/

News

NATO helicopter hovers over Intercontinental hotel

Taliban launch brazen attack on Kabul hotel

Updated: 28 June 2011. 23:58:40 (AEST)

NATO helicopters blasted Taliban militants holding out on the roof of a burning Kabul hotel this morning, after a brazen night-time attack which left an unknown number of people dead. >more

TV

At the Movies with Margaret and David

29/06/2011

Margaret and David review 'Transformers: Dark of the Moon', 'Cars 2', 'Mr. Popper's Penguins', 'The Trip' and Margaret's DVD Classic is 'Les Enfants du Paradis'.

Watch Now | Programs | TV Guide | Connect | Channels

Best of ABC – Our picks of unmissable content

Killing Fields survivors

Survivors of the killing fields (ABC News - Four Corners)

Refugees from Pol Pot's murderous regime reflect on the lives they've made for themselves in Australia.

Breathing botox

Severe asthma could be treated with injections of botox into the voice box.

Counselling controversy

School chaplains acting as unqualified counsellors could pose a risk to students.

Survival of the fibbest

Humans are hard-wired to be deceptive, says author Ian Leslie.

The Drum – Analysis and views on the issues of the day

Poll: Are net poll respondents more likely to vote "No" to any given question?

Reopening Manus will complicate PNG-Australia ties

Complicated ties

Ongoing political and legal wrangling over Manus and asylum seekers won't help relations between Australia and Papua New Guinea. >more 'Complicated ties'

The motherhood choice

I can't grasp why someone who wants to but can't have children feels that this state of affairs ranks as a tragedy that must be fixed at all costs. >more 'The motherhood choice'

Web of denial

No doubt there is a climate change debate and no doubt, either, that it is being manufactured. >more 'Web of denial'

Children & Parenting

Peppa Pig

Follow the adventures of a cheeky and slightly bossy little pig called Peppa every day at 5.05pm on ABC 4 Kids. >more 'Peppa Pig'

ABC 4 Kids | ABC3 | Parenting | Games | Kids TV guide

Grandstand Sport

Cleary to leave Warriors for Penrith

Ivan Cleary will leave the Warriors at the end of the NRL season to take on the head coaching position at Penrith for the next three years. >more 'Cleary to leave Warriors for Penrith'

Health Science

Apps the way to go with brain injury rehab

Digital helper Portable touch-screen technology can help patients overcome memory loss due to brain injury. > more 'Apps the way to go with brain injury rehab'

News in Science | Topics | Health News | Topics

ABC Open

Reflections on my past and my karma

At any moment a tragic turn of events can change a life forever. The great challenge is to emerge from tragedy with strength and grace. ABC Open contributor Ron Ipsen has faced that challenge and he shares his story with ABC Open.>more 'Reflections on my past and my karma'

Projects | Regions | Posts | About

May	JUN	Jul
	29
2010	2011	2012

Best of ABC – Our picks of unmissable content

The Drum – Analysis and views on the issues of the day

NoticeBoard

Featured Sites