Newsagent/Tech
Newsagent | |
Author | Chris Page |
---|---|
Contact | Support |
Repository | github |
Issue tracker | github |
OS | Neutral (web based) Will not work with IE8 or older |
URL | Newsagent |
Navigation | |
|
Importer
This section contains some notes regarding the importer system and the steps importer modules should follow.
General structure
The importer consists of a view/controller entrypoint module import
(Newsagent::Import
in blocks/Newsagent/Import.pm
), a base model module Newsagent::Importer
in modules/Newsagent/Importer.pm
. Newsagent::Importer
dynamically loads subclasses (of the form Newsagent::Importer::<SourceName>
, and here simply referred to as "sources" as a shorthand) to handle the importing of articles on a source-specific basic. Dynamic loading is done using through the system-wide Webperl::Modules::load_module()
, and the source modules must implement the import_articles()
method to do any work they need to do.
Invocation and execution flow
The importer is executed by loading the import
module - no auth is needed for this module, and it can be used via http or https: https://newsagent.cs.manchester.ac.uk/import/<source>
Here source can be one of:
all
- indicates that the importer should load and run all sources that need to run.- a source name, corresponding to a row with a given
shortname
in thenews_import_sources
table to run a specific source's import.
The import
module takes a number of steps:
- it checks the source is valid
- if the source if valid, it checks whether it is time to run the import for that source yet
- if it is time, the source module is loaded and its
import_articles()
is called.
(when the <source>
argument is all
, the above three steps happen for each defined source)
|