Unified Logging Service

We've created a subsystem of the simulator to provide a real time monitor of student interactions with the system for instructors to view to facilitate assessment of student actions.

We call it a "unified" logging service because it combines logs from several different sources into a single log. Currently it combines input from:

• CAD comment logs (generated by CAD server)
• CMS messages (generated by CPTMS)
• Activity Logs (generated by Activity Logger web app)

and in the future

• HAR messages (generated by CPTMS)
• Instructor Comments & Evaluations (generated by EI Notebook)

The output is a CSV file, unifiedlog.csv, which merges all the previous logs and orders them by simulation time (which it reads from sim_elapsedtime.json).

A separate web app, unifiedlogmonitor.html, runs in a web browser to display a nicely formatted version of the CSV file.

See attached Data Flow Diagram.

Configuration

A single configuration file, loggingservice.cfg, resides in the simulation config folder.

[Paths]
UnifiedLogPath = webapps/dynamicdata/
ActivityLogPath = webapps/dynamicdata/
[Files]
ActivityLogDataFilename = data.json
ActivityLogSummaryFilename = data_summary.json

The UnifiedLogPath option is to specify the path to the folder in which the CADcomments.log input file exists, and the output CSV file is to be created. Perhaps this isn't necessary; I can't think of a situation where this would change. Similarly, the ActivityLogPath specifies the path to the folder in which the Activity Log ("data.json") file exists. (This is "activitylog" on production system.)  When deployed to the web server, this path will be different than when testing on a local server. The Files section specifies the filenames of the activity log files, as they are produced by Neil's Activity Log.

Optionally, application_properties.cfg can be placed in config folder. For example,

[Versions]
revision_number = 452

Also, confirm that the CAD server properties file config/cad_simulator_config.properties has the correct location for writing the CAD comments log file:

# The location where the Coordinator writes the CAD comments log
CADcommentsLog         = webapps/dynamicdata/CADcomments.log

For production on the apache server, this path should point to W: drive, for example, W:\\webapps\\dynamicdata\\CADcomments.log

Packaging

The source code is in the src/python/unifiedlogger folder. The package.sh script found in that folder should be run to package the source files into a zip file and place it in the deploy folder. (Update 2022.9.1: package.sh is missing.)

Testing

Attached to this page are test data files: CADlogTestdata.json, CMSlogTestdata.json, ActivitylogTestdata.json and simtimeTestdata.txt.

Save the first one in the trunk folder and rename it to CADcomments.log

Save the second one in the webapps/dynamicdata folder and rename it to cms_messages.json

Save the third one in the webapps/dynamicdata folder and rename it to data.json

Save the fourth one in the webapps/dynamicdata folder and rename it to sim_elapsedtime.json

Open a terminal in the trunk folder and start the application: python deploy/unifiedlogger.zip

The application will echo logs to the console for diagnostic purposes. The expected output file is unifiedlog_expected.csv (attached). It should match the actual results in unifiedlog.csv.

While the application is running you may append further entries to either data file, save it, and observe the output csv file is updated with the new entry.

The unified logger updates the log every five seconds.

Open webapps/unifiedlogmonitor.html in a web browser and observe the csv file listed in reverse chronological order, formatted like this:

0:08:04	CAD log	        Incident #181	           Henry: HELLO FROM EARTH
0:08:04	CMS Activated.	N I-405 15.18 MAGNOLIA ST  'SLOW FOR THE::CONE ZONE:::'

This display is refreshed every five seconds.

When testing is complete, suspend the unified logger with Ctrl-Z and then kill the process with unix "kill" command.

Usage

Start the CAD server and Simulation Manager. Load the practice script and start it running. Start a CAD client. In a browser, start CPTMS. In a browser, start unifiedlogmonitor.html.

Wait for an incident to appear in the CAD client. Open the incident viewer and in the upper comments field, enter a note "hello world" and press Exit/Send. Wait five seconds and observe an entry for CAD Log appear in the monitor.

In CPTMS, display the CMS layer, open a CMS dialog and enter a message "SLOW TRAFFIC". Click "Send" and wait five seconds. Observe an entry for CMS Activated to appear in the monitor. Update the CMS message results in a "CMS Updated" entry in the unified log. Clearing a CMS message results in a "CMS Deactivated" entry in the unified log.

Deploying on Apache Server

Copy unifiedlogger.zip and unifiedlogmonitor.html to webapps folder on W: drive.
Copy loggingservice.cfg and application_properties.cfg to config folder (at same level as webapps folder).
Start the application: python webapps/unifiedlogger.zip

Running as a background service

TBD: Explain how to start the service as a unix daemon. Explain start, stop, restart. Explain which files are reset, etc.