What is Combine?

A Focused Crawler System for the WEB

If you want to download
- all Web-pages from a list of servers (like all servers at Lund University)
or
- all Web-pages pertaining to a particular topic (like 'Carnivorous Plants')

Then Combine focused crawler is the system for you!

Combine is an open system for crawling [harvesting and threshing (indexing)] Internet resources. It can be used both as a general and focused crawler. The name is derived from the combine-harvester since the two perform their jobs in a similar way.

Its is implemented in Perl and is easily configured for most types of Web-crawling.

Together with the Zebra text indexing and retrieval engine Combine makes an easy to use search engine in a box. In a few simple steps you create a vertical search engine with structured searching.

Features

• part of the SearchEngine-in-a-Box system
• higly configurable
• integrated automated topic classifier for focused crawling mode
• possibility to use any topic classifier (if provided as a Perl PlugIn module) in focused crawling mode
• crawl based on regular expression on URLs both include and exclude
• characterset detection/normalization (UTF-8)
• language detection
• HTML cleaning
• metadata extraction
• many document types (text, HTML, PDF, PostScript, MsWord, PowerPoint, Excel, RTF, TeX, images)
• obeys robot exclusion
• SQL database for data storage and administration

Who is using Combine?

Services, demos

The Combine Robot was first released (spring '98), and was primarily used whithin the nordic NWI services (out of commision). A modified version of Combine was also used in the Kulturarw Web archival project.

We envisage it as being useful for regional web indexing in general.

SOSIG use Combine as harvester for their Social Science Search Engine, based on urls from the main catalogue.

In the EU project ALVIS it is used (and further developed) as a focused crawler in order to produce data collections for the topic-specific semantic-based search engine that ALVIS develops. There are some ALVIS demos are available.

Many of KnowLib demos use Combine crawled data.

You may want to use it yourself? That is possible, since it is all built upon free software. See Downloads below

Where can I get further information?

Documentation

• Examples, HowTo's and hints.
• Detailed documentation, man pages (updated 2006-12-08)
• Publications
• Modifications to work as a P2P distributed network
• Simulation tool to study focused Web crawling strategies including downloads of software

Mailing list at SourceForge

Installation

Installation Debian stable

• Add the line
  deb http://combine.it.lth.se/ debian/
  to your file /etc/apt/sources.list
• apt-get update
• apt-get install combine
  this also drags in MySQL and a lot of Perl modules

Installation from CPAN

Manual installation for the impatient

perl Makefile.PL
make
make test
make install
mkdir /etc/combine
cp conf/* /etc/combine/
mkdir /var/run/combine

Test that it all works (run as root)
perl ./doc/InstallationTest.pl

Test the installation

• sudo combineINIT --jobname aatest (Must be run as root)
  will create a MySQL database called 'aatest' plus create and initialize the job specific configuration directory '/etc/combine/aatest/'
• enter some (at least one) seed URLs like
  echo 'http://www.eit.lth.se/' | combineCtrl --jobname aatest load
• combineCtrl --jobname aatest start
  will start 1 crawler process
• combineCtrl --jobname aatest stat
  will give status of the frontier que (the list of URLs currently beeing crawled).
• combineCtrl --jobname aatest kill
  kills all running crawlers for the job 'aatest'
• combineExport --jobname aatest --profile combine
  will export to STDOUT all records for job 'aatest' stored in the MySQL database. It will be formatted in XML.

Where can I get help?

For reporting bugs, getting help or information or just to let us know that you are using Combine, send an e-mail to Combine general discussion list which is a mailing list at Subscribe to the list here.

Downloads

• SearchEngine-in-a-Box system
• a Debian repository (also includes some necessary Perl CPAN modules not available in the main Debian release). Add the line
  deb http://combine.it.lth.se/ debian/
  to your file /etc/apt/sources.list

  Source for version 4.003 (gzipped tar)
  

  (Using the debian package-system - apt - is much more streamlined and easier to use. Debian packages contain dependencies, and automatically install all dependendent packages, see above under 'Installation'.)

• Simulation tool for URL scheduling algorithms in focused Web crawling (see tool page)
  • Source code for Simulation tool (Java)
  • Post processing utilities in Matlab

• P2P distributed version Highly experimental.
• Versions 1.x

It is later beeing modified for focused crawling by integrating the automated topic classification algorithms also developed in DESIRE with the crawler. This work was funded by Vinnova, Swedish Agency for Innovation Systems (project P22504-1 A) and the EU project ALVIS project (IST-1-002068-STP). Currently supported by .SEs Internetfond in the project 'Vertical Search Engines'.

Copyright

Last modified
2009-10-23