Information extraction agent

Abstract

A time has come upon us where the world of information has grown beyond anything we have ever imagined it would be. Due to the creation of the internet and the World Wide Web, we now possess vast amounts of information stored in repositories of varying representations and formats whose collective mass exceeds the current methods of modem searching and retrieval. The users of the web are often lost in these huge realms of information and the gathering of the smallest and most trivial knowledge is too frequently too time-consuming. With the progress made in the area of human natural language processing software, together with other artificial intelligence methods, a relatively new study of information gathering has emerged which identifies particular pieces of information within large unstructured texts, acting as an intelligent search on our behalf and referred to in the computer science domain as 'Information Extraction'. This thesis is all about this intelligent means of extraction from within pages of the World Wide Web and describes the design and development of a system built to extract information from HTML based free text articles found within newspaper web pages. This system, based on the principles of information extraction and referred to as an 'Information Extraction Agent', endeavors to create an automatic means for web users to find relevant articles based on their criteria from these data repositories which cannot be traversed through structured queries and database instructions. Therefore, the system requires algorithms which attempt to "understand'' the facts lurking within these articles in order to be able to deduce which articles are useful to a given particular user.