global_lib api documentation

class global_lib.replica_engine(args)

Bases: object

This class is wraps the the mysql and postgresql engines in order to perform the various activities required for the replica. The constructor inits the global configuration class and setup the mysql and postgresql engines as class objects. The class sets the logging using the configuration parameter.

add_source()

The method adds a new replication source. A pre existence check is performed

copy_schema()

The method calls init_replica adding a flag for skipping the data copy. Useful if we want to test for schema issues or to populate the schema preventively.

create_replica_schema()

The method creates the replica schema in the destination database.

detach_replica()

The method terminates the replica process. The source is removed from the table t_sources with all the associated data. The schema sequences in are reset to the max values in the corresponding tables, leaving the postgresql database as a standalone snapshot. The method creates the foreign keys existing in MySQL as well. Is compulsory to specify a source name when running this method.

drop_replica_schema()

The method removes the replica schema from the destination database.

drop_source()

The method removes a replication source from the catalogue.

enable_replica()

The method resets the source status to stopped and disables any leftover maintenance mode

init_replica()

The method initialise a replica for a given source and configuration. It is compulsory to specify a source name when running this method. The method checks the source type and calls the corresponding initialisation’s method.

load_config()

The method loads the configuration from the file specified in the args.config parameter.

read_replica(queue, log_read)

The method reads the replica stream for the given source and stores the row images in the target postgresql database.

refresh_schema()

The method reload the data from a source and only for a specified schema. Is compulsory to specify a source name and an origin’s schema name. The schema mappings are honoured by the procedure automatically.

replay_replica(queue, log_replay)

The method replays the row images stored in the target postgresql database.

run_maintenance()

The method runs a maintenance process on the target postgresql database specified in the given source.

set_configuration_files()

The method loops the list self.conf_dirs creating them only if they are missing.

The method checks the freshness of the config-example.yaml and connection-example.yml copies the new version from the python library determined in the class constructor with get_python_lib().

If the configuration file is missing the method copies the file with a different message.

show_config()

The method loads the current configuration and displays the status in tabular output

show_errors()

displays the error log entries if any. If the source the error log is filtered for this source only.

show_sources()

The method shows the sources available in the configuration file.

show_status()

list the replica status from the replica catalogue. If the source is specified gives some extra details on the source status.

start_replica()

The method starts a new replica process. Is compulsory to specify a source name when running this method.

stop_all_replicas()

The method stops all the active replicas within the target database

stop_replica()

The method calls the private method __stop_replica to stop the replica process.

sync_tables()

The method reload the data from a source only for specified tables. Is compulsory to specify a source name and at least one table name when running this method. Multiple tables are allowed if comma separated.

terminate_replica(signal, frame)

Stops gracefully the replica.

update_schema_mappings()

The method updates the schema mappings for the given source. The schema mappings is a configuration parameter but is stored in the replica catalogue when the source is added. If any change is made on the configuration file this method should be called to update the system catalogue as well. The pg_engine method checks for any conflict before running the update on the tables t_sources and t_replica_tables. Is compulsory to specify a source name when running this method.

upgrade_replica_schema()

The method upgrades an existing replica catalogue to the newer version. If the catalogue is from the previous version

class global_lib.rollbar_notifier(rollbar_key, rollbar_env, rollbar_level, logger)

Bases: object

This class is used to send messages to rollbar whether the key and environment variables are set

send_message(message, level)

The method sends a message to rollbar. If it fails it just logs an error without causing the process to crash.