-2

I'm currently developing a web crawler. The first version was developed in Node.js and runs pretty well.

The issues that I encountered with Node.js are in no particular order:

  • slow URL and query-string parsing library
  • slow HTTP packet parsing due to the C++ binding call overhead
  • slow DNS resolution due to synchronous call to getaddrinfo rather than the asynchronous C-Ares library
  • lack of good Unicode support in RegExps

These are for the most part due to the fact that it's still very young and that it's a general purpose language with a small core developer team.

The proof-of-concept I have at the moment runs pretty well thanks to the asynchronous nature of Node.js and some monkey-patches that I had to add.

Now I'm starting to think about what comes after the proof-of-concept.

The question of the programming language is one I wasn't able to answer yet.

The essential qualities that I'm looking for:

  • I/O and concurrency management
  • fast string parsing capabilities for HTTP messages, HTML, JSON ...etc.
  • bindings to some kind of queuing / persistence layer

I thought about C, Erlang, Rust, Go and D.

I know this isn't usually the kind of questions that are accepted here on Stack Exchange but I'm not sure where else I could get an answer.

Update: Some more information:

  • we currently use 8 core, 3.2 Ghz machines.
  • the persistence layer is ensured by running a PostgreSQL database
  • the point is to crawl all kinds of news sites and blogs as well as RSS feeds
  • we currently forward the content to a processing pipeline through Apache Kafka
Improve this question
9
  • You could also consider Common Lisp, Haskell, Ocaml. How do you run your crawler: do you have a single-core machine or an entire data-center or supercomputer with thousands of multi-core processors? You probably won't use the same kind of language in those different cases... Commented May 1, 2015 at 12:41
  • @BasileStarynkevitch: We currently run 8 core machines with one process per core. Commented May 1, 2015 at 12:44
  • You should edit your question, instead of commenting it. Why are you crawling, what for? Do you have a single database? How are you managing indexing? Are you interested in multi-threading, or do you just have several processes? How are they communicating? Commented May 1, 2015 at 12:47
  • An important tradeoff is developer's time vs computer time. If it is an academic or single short experiment, you care more about ease of development than about raw performance. If on the contrary you hope to extend your crawling to thousands of computers, you probably can afford to spend more development efforts. BTW Clojure or Scala are probably also worthwhile. Also, the choice of the programming language is related to the developers you have (or can afford), and I would leave that choice to developers. Commented May 1, 2015 at 12:53
  • 1
    @m_vdbeek I am voting to keep this question closed because it is still not a good question. It is on the border of "too broad" and "primarily opinion based." It needs to be more specific and objectively answerable. Also as an FYI, if you edit a question that is either "on hold" or "closed" it goes into a review queue where 3k rep users and moderators can vote to reopen. No need to ask for it. Commented May 1, 2015 at 13:40

1 Answer 1

Reset to default
3

I've developed a crawler in Python for educational purposes (TripAdvisor Scraper).

It's built upon Scrapy for crawling the web and I'd choose Python because it is good for Natural Language Processing techniques, you have a lot of toolkit and resources (NLTK). Python is not so fast to process high volumes of data, and you have to deal with memory management to keep things fast. Scrapy isn't able to run JavaScript on web pages, so if you have something that changes at runtime via JS, you won't be able to retrieve it.

Now I'm writing a full asynchronous system, able to scale out and self repair, for crawling data from various sources.

I'm using Scala for functional programming (no side effect, easy to reach high parallelism) + Akka (actor based model, easy to isolate things and scale out the entire system. It provides supervisors strategies). Akka paradigm is strongly inspired by Erlang actor model system. To manipulate the DOM I'm using JSOUP, it's a full library to deal with HTML and it's able to execute JavaScript. So it's good to simulate real browser interaction. The entire stack is based upon JVM.

This new stack is extremely fast compared to the first one and it's reactive and resilient.

I would suggest to go with the second solution, or if you want to try I think that Erlang is good enough to build a fast crawler and easy achieve parallelism.

Improve this answer
3
  • I'm sorry, I'm not sure who down voted you for a perfectly valid answer ... When you parse HTML, is it as a stream / pipe or do you wait and buffer the entire body before sending it to JSOUP or other format parsers ? Commented May 1, 2015 at 12:24
  • We run all our NLP and machine-learning algorithms on Spark, so Scala in already present in our stack, which is nice. Commented May 1, 2015 at 12:32
  • Using JSOUP I'm using org.jsoup.Jsoup.connect().get() which I think that will act as a buffer to create a org.jsoup.nodes.Document object after retrieving the entire HTML response body (it'll throw an exception if something went wrong). There are also methods for using JSOUP with streams, org.jsoup.Jsoup.parseBodyFragment() and org.jsoup.Jsoup.parse(). I'm also using Spark for NLP. Regarding mine API layer I'm using Play while I'm waiting for akka-http. Commented May 1, 2015 at 15:03

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.