Newsagent/Tech

From School of CS Wiki
Jump to: navigation, search
Newsagent
Newsagent Logo.png
Author Chris Page
Contact Support
Repository github
Issue tracker github
OS Neutral (web based)
Will not work with IE8 or older
URL Newsagent
Navigation

Importer

This section contains some notes regarding the importer system and the steps importer modules should follow.

General structure

The importer consists of a view/controller entrypoint module import (Newsagent::Import in blocks/Newsagent/Import.pm), a base model module Newsagent::Importer in modules/Newsagent/Importer.pm. Newsagent::Importer dynamically loads subclasses (of the form Newsagent::Importer::<SourceName>, and here simply referred to as "sources" as a shorthand) to handle the importing of articles on a source-specific basic. Dynamic loading is done using through the system-wide Webperl::Modules::load_module(), and the source modules must implement the import_articles() method to do any work they need to do.

Invocation and execution flow

The importer is executed by loading the import module - no auth is needed for this module, and it can be used via http or https: https://newsagent.cs.manchester.ac.uk/import/<source> Here source can be one of:

  • all - indicates that the importer should load and run all sources that need to run.
  • a source name, corresponding to a row with a given shortname in the news_import_sources table to run a specific source's import.

The import module takes a number of steps:

  • it checks the source is valid
  • if the source if valid, it checks whether it is time to run the import for that source yet
  • if it is time, the source module is loaded and its import_articles() is called.

(when the <source> argument is all, the above three steps happen for each defined source)


Authors

  • gravatar Mbasscp2 [userbureaucrateditorhandbook-editstaff-editsysopPHRhYmxlIGNsYXNzPSJ0d3BvcHVwIj48dHI+PHRkIGNsYXNzPSJ0d3BvcHVwLWVudHJ5dGl0bGUiPkdyb3Vwczo8L3RkPjx0ZD51c2VyPGJyIC8+YnVyZWF1Y3JhdDxiciAvPmVkaXRvcjxiciAvPmhhbmRib29rLWVkaXQ8YnIgLz5zdGFmZi1lZGl0PGJyIC8+c3lzb3A8YnIgLz48L3RkPjwvdHI+PC90YWJsZT4=]