Unified Logging Service

We've created a subsystem of the simulator to provide a real time monitor of student interactions with the system for instructors to view to facilitate assessment of student actions.

We call it a "unified" logging service because it combines logs from several different sources into a single log. Currently it combines input from:

• CAD comment logs (generated by CAD server)
• CMS messages (generated by CPTMS)
• Activity Logs (generated by Activity Logger web app)
• HAR messages (generated by CPTMS)
• Instructor Comments & Evaluations (generated by EI Notebook)

The output is a CSV file, unifiedlog.csv, which merges all the previous logs and orders them by simulation time (which it reads from sim_elapsedtime.json).

A separate web app, unifiedlogmonitor.html, runs in a web browser to display a nicely formatted version of the CSV file.

See attached Data Flow Diagram.

Configuration

A single configuration file, loggingservice.cfg, resides in the simulation config folder.

[Paths]
UnifiedLogPath = webapps/dynamicdata/
ActivityLogPath = webapps/dynamicdata/
[Files]
ActivityLogDataFilename = data.json
ActivityLogSummaryFilename = data_summary.json

The UnifiedLogPath option is to specify the path to the folder in which the CADcomments.log input file exists, and the output CSV file is to be created. Perhaps this isn't necessary; I can't think of a situation where this would change. Similarly, the ActivityLogPath specifies the path to the folder in which the Activity Log ("data.json") file exists. (This is "activitylog" on production system.) When deployed to the web server, this path will be different than when testing on a local server. The Files section specifies the filenames of the activity log files, as they are produced by Neil's Activity Log.

Optionally, application_properties.cfg can be placed in config folder. For example,

[Versions]
revision_number = 452

Also, confirm that the CAD server properties file config/cad_simulator_config.properties has the correct location for writing the CAD comments log file:

# The location where the Coordinator writes the CAD comments log
CADcommentsLog         = webapps/dynamicdata/CADcomments.log

For production on the apache server, this path should point to W: drive, for example, W:\\webapps\\dynamicdata\\CADcomments.log

Packaging

The source code is in the src/python/unifiedlogger folder.  Use the script in trunk/bin/mkunifiedloggerzip.sh to package the source files into a zip file and place it in the deploy folder.

Testing

Attached to this page are test data files: CADlogTestdata.json, CMSlogTestdata.json, ActivitylogTestdata.json and simtimeTestdata.txt.

Save the first one in the trunk folder and rename it to CADcomments.log

Save the second one in the webapps/dynamicdata folder and rename it to cms_messages.json

Save the third one in the webapps/dynamicdata folder and rename it to data.json

Save the fourth one in the webapps/dynamicdata folder and rename it to sim_elapsedtime.json

Open a terminal in the trunk folder and start the application: python deploy/unifiedlogger.zip

The application will echo logs to the console for diagnostic purposes. The expected output file is unifiedlog_expected.csv (attached). It should match the actual results in unifiedlog.csv.

While the application is running you may append further entries to either data file, save it, and observe the output csv file is updated with the new entry.

The unified logger updates the log every five seconds.

Open webapps/unifiedlogmonitor.html in a web browser and observe the csv file listed in reverse chronological order, formatted like this:

0:08:04	CAD log	        Incident #181	           Henry: HELLO FROM EARTH
0:08:04	CMS Activated.	N I-405 15.18 MAGNOLIA ST  'SLOW FOR THE::CONE ZONE:::'

This display is refreshed every five seconds.

When testing is complete, suspend the unified logger with Ctrl-Z and then kill the process with unix "kill" command.

Usage

Start the CAD server and Simulation Manager. Load the practice script and start it running. Start a CAD client. In a browser, start CPTMS. In a browser, start unifiedlogmonitor.html.

Wait for an incident to appear in the CAD client. Open the incident viewer and in the upper comments field, enter a note "hello world" and press Exit/Send. Wait five seconds and observe an entry for CAD Log appear in the monitor.

In CPTMS, display the CMS layer, open a CMS dialog and enter a message "SLOW TRAFFIC". Click "Send" and wait five seconds. Observe an entry for CMS Activated to appear in the monitor. Update the CMS message results in a "CMS Updated" entry in the unified log. Clearing a CMS message results in a "CMS Deactivated" entry in the unified log.

Deploying on Apache Server

Copy unifiedlogger.zip and unifiedlogmonitor.html to webapps folder on W: drive.

Modify the entry in logging_service.cfg to reflect the location of activity log files on the production webserver, probably: ActivityLogPath = activitylog

Copy logging_service.cfg and application_properties.cfg to config folder (at same level as webapps folder).
From the webserver home directory, start the application: python webapps/unifiedlogger.zip

Running as a background service

TBD: Explain how to start the service as a unix daemon. Explain start, stop, restart. Explain which files are reset, etc.

Sample output

00:00:00, CAD Log, Incident #181, Detail: ENRT FROM NB 405 AT JEFFREY
00:01:18, Activity Log,,021 10-98 Assignment Completed 75-1
00:01:18, Activity Log,,021 10-98 Let's get pizza. 75-1
00:01:18, Activity Log,Bailey Mary,001  Incident Created
00:01:18, Activity Log,Gross Angelo,001  32-1 Vehicle Disabled S 55@ 405 1308 hours
00:01:18, Activity Log,Gross Angelo,001  HCC NOTIFIED
00:01:18, Activity Log,Ha Khanh Duy,001  Per 32-1 NB rte 405 blocked
00:01:18, Activity Log,Hockaday Neil,002  Incident Created
00:01:18, Activity Log,Quan Ally,002  Incident Created
00:01:18, CAD Log, Incident #187, Jose Trainee: HELLO TMC WORLD
00:01:18, CAD Log, Simulation Started at 26 Dec 2019 16:40:07 with incidents 100 187 188 189 190 191
00:01:18, CMS Activated,N I-405 15.18 MAGNOLIA ST, 'SLOW FOR THE::CONE ZONE:ROAD WORK:AHEAD:'
00:01:18, CMS Activated,N SR-55  5.38 Baker Street, 'BBBBB:::::'
00:01:53, CMS Activated,N SR-73 24.91 North of Jamboree, 'LINES ON ROAD:::::'
00:02:28, CMS Updated, N SR-73 24.91 North of Jamboree, 'SNOW ON ROAD:::::'
00:03:00, CAD Log, Incident #181, Detail: 14-13 10-97 1125  EMS 10-97  HAY TRK VS 2 VEH  #1 2 3 LNS BLCKD REQ ADDL UNITS
00:03:13, CAD Log, Incident #187, Jose Trainee: SEND ICE CREAM
00:03:33, CMS Deactivated, N SR-73 24.91 North of Jamboree
00:03:45, HAR Activated, KNNN868 1620 AM,'TRAFFIC DELAYS ON 405 SOUTHBOUND'
00:04:00, CAD Log, Incident #181, Detail: 14-7 ENRT FROM SB 405 AT BROOKHURST 14-21 ENRT FROM 55 AT MACARTHUR
00:04:04, CAD Log, Simulation Paused.
00:05:00, CAD Log, Incident #181, Detail: 1144 IN HAY TRK  2 1181S  REQ CORONERREQ DOT TO REMOVE HAY SPILLED IN SB LANES
00:07:00, CAD Log, Incident #181, Detail: 14-7 REQ 1185 BIG RIG AND 2 ROT 1185
00:09:00, CAD Log, Incident #181, Detail: 10-39 COLLEGE OAK FOR BIG RIG AND CYCLE FOR ROT
00:11:00, CAD Log, Incident #181, Detail: 14-21 1097 PROV 1184 TRF BACKING QUICKLY
00:12:00, CAD Log, Incident #181, Detail: VEH KNOCKED HAY BALES ONTO VON KARMAN AVENUE ADV CITY
00:15:00, CAD Log, Incident #181, Detail: 1181 TRANSPORTED REQ ETA FOR CORONERETA FOR CORONER 10
00:18:00, CAD Log, Incident #181, Detail: 14-13 1185 1097  #3 LN WILL OPEN 5-10
00:20:00, CAD Log, Incident #181, Detail: DOT 1097 TO REMOVE BALES
00:23:00, CAD Log, Incident #181, Detail: CORONER 1097
00:24:00, Evaluation, FACILITATOR, 'anything will do'
00:24:00, Evaluation, FACILITATOR, 'snoopy says go'
00:24:00, Evaluation, FACILITATOR, 'we all scream for ice cream'
00:24:00, Evaluation, RADIO, 'Forgot  pizza toppings'
00:24:00, Evaluation, RADIO, 'Had trouble finding phone number'
00:24:00, Evaluation, RADIO, Poor
00:25:00, CAD Log, Incident #181, Detail: #3 LN CLEARED OPENING NOW
00:25:00, Evaluation, ACTIVITY_LOG, 'User prefers mushrooms over peppers'
00:25:00, Evaluation, FACILITATOR, 'When out at night, where white'
00:28:00, CAD Log, Incident #181, Detail: CORONER CONFIRMED 1144  VEHS READY FOR TOW  HAY NEARLY CLEARED  ALL LANES OPEN IN 15
00:30:00, CAD Log, Incident #181, Detail: WTNSSES  JOHN/BETTY JONES SAW HAY TRUCK LOSE CONTROL  FLIP OVER  2 VEH HIT
00:32:00, CAD Log, Incident #181, Detail: HAY TRK READY FOR TOW  2 VEHS TOWED  DOT CLEARED HAY BALES  LANES OPEN IN 10
00:35:00, Evaluation, ACTIVITY_LOG, 'every good boy does fine'
00:36:00, CAD Log, Incident #181, Detail: LANES OPEN IN 5
00:36:00, Evaluation, FACILITATOR, 'Prefer chocolate ice cream in the cone'
00:36:00, Evaluation, FACILITATOR, 'Put cones on heads'
00:41:00, CAD Log, Incident #181, Detail: 14-13 14-7 14-21 1098  TRK TOWED  ALL LNS CLR
01:38:00, Evaluation, CMS, Worst
01:43:00, Evaluation, ACTIVITY_LOG, Best
01:43:00, Evaluation, ACTIVITY_LOG, Worst
01:44:00, Evaluation, ACTIVITY_LOG, Good
01:46:00, Evaluation, ACTIVITY_LOG, Average
01:53:00, Evaluation, ATMS, Good
01:53:00, Evaluation, CAD, Best
01:53:00, Evaluation, CAD, Poor
0:24:41, LCS new closure: Sue Trainee, T133AA.1, 133EB, Lane, Special Event
0:24:41, LCS new closure: Sue Trainee, T1AC.1, 1SB, Full, AC Paving
0:24:41, LCS new closure: User, T5AD.1, 5NB, Lane, Drainage Inspection
0:24:41, LCS status update: Sue Trainee, T73AA, 1097, Marjie 
0:24:41, LCS status update: Sue Trainee, T1AB, 1022, JeNnY 
0:24:41, LCS status update: Sue Trainee, T22AA, 1022, Joel 
0:24:41, LCS status update: Sue Trainee, T22AB, 1022, april may 
0:24:41, LCS status update: Sue Trainee, T22AB, 1098, APRIL MAY 
09:03, LCS status update: Sue Trainee, T22AA, 1098, JOEL 
09:31:46, LCS status update: Sue Trainee, T1AB, 1097, Jenny 
09:36:26, LCS status update: Sue Trainee, T1AB, 1098, JENNY