SiteFX Quick Tips

Using SiteFX to Test for Logger File Compatibility 

PWQMN Data Loading 

Using SiteFX to Test for Logger File Compatibility

SiteFX’s data logger file importer differs from the general purpose data loader in that, 1) the logger file header must first be decoded and 2) logger data often requires calibration and a ‘reality check’ before the data is added to the database.  Logger file formats are a dime a dozen - this tip explains how to test for logger file compatibility before using the importer.

A snippet from an example logger file is shown below:

In this particular case, the file is a .prn text file with about a dozen lines for the header followed by the actual data.

Follow the steps below to check for logger file compatibility.

Open up the main form for loading data logger files by going to Temporal Data > Data Loggers > Import Logger Data...

Once the form that opens, click the Check Format button in the top right corner:

A new form will appear called File Compatibility Check (shown below). Press the Select Files button and select one or more files (If several files are selected, the tool will provide a quick report on one file at a time). An example of a file that has been checked is shown below:

In the summary text in the screenshot above, the most important lines have been outline in red.

“Is File Format Recognized?” refers to whether SiteFX understands the header or not.

“Is All Data Readable?” means that SiteFX has read through the entire data section of the logger file and checked for inconsistencies in format.

If the first two are both true, your file is loadable.  If not, contact EarthFX to request support for that file format.

The third line, “Matching Logger Found in Database?” is true or false depending on whether you have linked this logger name to a well in the database yet.  This linking is done while editing the location in the location properties editor (accessed from the Location Browse & Search tool).

PWQMN Data Loading

Users with project work in Ontario, Canada, can use SiteFX's PWQMN (Provincial Water Quality Monitoring Network) data loader to load data from the PWQMN - a network of almost 400 surface water locations across Ontario. The SiteFX PWQMN data loader ensures importing of these datasets is as efficient and consistent as possible.

A snippet of a PWQMN data file is shown below:

Assumptions

The following assumptions are made:

• Each record must have a date. The date must be 10 characters in length and formatted as YYYY-MM-DD as in the example snippet above.  In Excel, these cells will be formatted with the ‘General’ formatting type.
• All times must have between 1 and 4 characters. Blanks are not allowed. (Note: 1 character is allowed because a time of ‘0’ can appear in the PWMQN file)
• If the value in the TIME column is a 0 then 12:00 AM is assumed. (Note: Using 12:00 AM essentially retains the fact that the actually sampling time is unknown.)
• A REMARK of ‘000’ means ‘no remark’ and it is treated the same as if It were blank. All other remarks are imported as value qualifiers.

Processing Notes

Duplicate Records

A very important check has been put in place in order to prevent the same data from being loaded more than once. This check occurs at the record level, which means that if part of the file contains new data and part of the file contains duplicate data (data that has already been loaded) only the new data will be imported.

As a result, SiteFX will prevent the same file from being loaded twice AND handle the situation where several PWQMN data files contain some overlap of data.

The check for duplicate data uses the following fields: STATION ID, PARAMETER DESCRIPTION, DATE, TIME, RESULT and UNITS. (Note: REMARK is purposely omitted from the duplicate check.)

Unit Conversions

Units are converted according to the unit conversions library that can be edited and maintained by the user. These conversions can be managed before OR after the data has been loaded. (Note: the original data values and units are always stored in the database. Subsequent conversions always refer back to the original data values)

Parameter Descriptions

Parameter names are mapped to the spelling and/or abbreviation of the user’s choice according to the parameter library that can be edited and maintained by the user. The parameter name library can be managed before OR after the data has been loaded. (Note: the original parameter name is always stored in the database.)

Accessing the PWQMN Data Loader

SiteFX understands a variety of data formats and the PWQMN Excel format has been added to this library.  The data loader is accessed from the Temporal Data menu as shown in the screenshot below:

When the ‘Load Data File’ option has been selected from the menu, the following form will appear:

Clicking on the ‘Load File’ button will prompt the user to select one file. (Note: only one file can be loaded at a time). Upon selection of a file, the Data Loader form will update and appear similar to the screenshot below and the user will be notified when the importing has been completed.

Other Considerations

Duplicate Records

In the event that data in the import file is already in the database, the user will be notified of the number of duplicate records and asked how they would like to proceed. An example notification is shown below.

Parameter Names

The user’s database contains a parameter name library which is a list of all parameter names and alias names being used by the database. If during import SiteFX encounters a parameter name that has not been seen before by the database, SiteFX will automatically add the parameter name to the library if the user has set that option under Tools > Options. If the option has not been set, they will be able to either add the parameter names later, or map the names to existing parameter names (e.g., the spelling has changed).

Loaded File Manager

To view a list of files loaded by SiteFX go to Temporal Data > Manage Loaded Data Files. This tool is useful to see what files have been loaded and, if necessary, View or Delete the data. Note that the ‘View’ option shows just the full dataset and does not come close to the functionality provided by the Analyze Data tool, which is used for graphing and reporting. A screenshot of the Manage Loaded Data Files tool is shown below: