5.9.3 Data loader

The data loader periodically checks the tmp directories for every data collector module and looks for new SQLLDR files. It waits 15 seconds between each cycle. If a new file has arrived then it downloads it to its sftp directory and updates its status table in the database. If the value of the USE_PARTITONS parameter in the CONFIG_INI.pm file is less than 2 then it loads the file as it is into the database and updates the status table in the database again. However if the value of the parameter is >= 2 then it first tries to merge data files together, to be able to load a bigger file into the database in one turn. If a collector server group has more than one collector server then the module will not load the result file from a collector server into the database until all files from all collectors belonging to this group have arrived (or it determines that too much time have passed and will not wait longer). If the value of USE_SORT_FOR_UNIQUE_MERGE is set to 1 then it sorts the data file before loading it and also eliminates duplicated rows: this can happen if a collector group has more than just one collector server, so in cases like this it is recommended to set this parameter to 1.

If every file for a particular measurement interval has been loaded into the database or it knows that it shouldn’t wait longer than it closes the data loading for that interval / time pair, i.e. it updates the status table in the database. The report and the threshold processing starts a new cycle only when this has happened. The files downloaded from the collector modules end up in one of the following directories after they have been processed by the SQLLDR module:

·       tmp/done: if the file was loaded into the database and there was no error during the loading

·       tmp/imperror: if the file was loaded into the database and but there was an Oracle error during the loading

·       tmp/skipped: if the file was skipped because SQLLDR has already processed the date&time this file is referring to

PVSR only deletes files automatically from the tmp/done directory after a while.

The module performs its self-checkings every five minutes, see the Server events section.

By default the data loader consists of two processes: a main process which starts or restarts the child process and child process which does the actual processing but terminates automatically once every day. After the termination the main process starts it again automatically. The module might spawn other child processes based on the value of the SQLLDR_ASYNC_LDR_MODE configuration parameter, which determines how the module looks for new files and how it loads them into the database. The parameter has four possible values:

·       0 (default): the module downloads the measurement result files one-by-one and then loads them into the database

·       1: the module downloads the measurement result files one-by-one but while loading them into the database it keeps looking for new result files and keeps downloading them

·       2: the module forks new processes and downloads the measurement result files parallel and then loads them into the database

·       3: the module forks new processes and downloads the measurement result files in parallel and while loading them into the database it keeps creating new processes and keeps downloading the result files parallel

For the tables used during data loading and available storage options see the “Measurement result storage options” section.