Lessons Learned
Table of Contents
Purpose
The purpose of this document is to describe the problems we were trying to solve in developing the Traffic Model and FEP Simulator and the obstacles we encountered during design and development.
Background and Call to Action
In previous iterations of the TMC Simulator system, a commerical microscopic traffic modeling software named Paramics was used. UC Irvine wrote plugin code to interface with Paramics. Paramics was causing technical issues as time went on and systems evolved, and was a costly commercial solution. It was decided that the best option was to create our own traffic modeling system, which would meet our systems requirements and give us the benefit of owning and controlling the code for future development.
Initial Plan
Initially, the idea was to create a self-driving macroscopic traffic model. In other words, a traffic model that simulated incidents without user interaction. Little to none was known about the ATMS at this time, and the FEP Simulator and Traffic Model had not been designed.
Challenges
Gathering Resources and Documentation for Existing System Components
Gathering resources and documentation for the existing system components, such as the ATMS and FEP, proved to be difficult and took more time than expected. Also, the documentation received had a lot of extra fluff. It took a considerable amount of time to gather the information from developers and CalTrans. The UCI Paramics plugin code's location was unknown. We never received any documentation on the ATMS, which made design and development particularly difficult. Luckily, we were able to get enough information from the FEP documentation, UCI Paramics plugin code, and files on the ATMS to make things work.
Another challenge in documentation for the existing system was that there was a lot of extra fluff. It took a considerable amount of time to find what we needed out of the immense amount of technical documentation for our system. For example, the FEP documentation was the documentation for the production FEP, so it included a lot of information that was not pertinent to our simulation system. We also received about three our four different versions of the FEP documentation, some versions relevant, some not. Deciphering through these docs took a great deal of time and effort.
RPC Calls and the Birth of the FEP Simulator
A challenge that we faced was the use of RPC. RPC is not the most integrable technology around. Unfortunately for us, Java does not support RPC but our entire simulation system was written in Java.
For this problem, we saw three possible solutions:
- Write an intermediary program on a linux system in C, which reads data from the Java CAD Server and then interfaces with the ATMS via RPC.
- Purchase the ONC library to use RPC with Java.
- Write our own Java RPC library using Java Native Interface methods.
We chose to write an intermediary program on a linux system in C, which read data from the Java CAD Server and then interfaced with the ATMS via RPC. This gave us ownership of the code, independence from the external commercial ONC library and its financial expense, less development time than writing our own Java RPC library using JNI, and a more realistic simulation of the FEP.
Interestingly enough, this "intermediary program" became the FEP Simulator. This was not our intent at first, we just wanted something that could communicate with the ATMS via RPC! After developing this program some, we realized that it essentially modeled the FEP. Hence, we named it the FEP Simulator.
RPC Platform Problems
When developing the FEP Simulator, we did so on a Linux operating system which supports RPC natively. We did not realize that putting our program on our production computers, which run a Windows operating system, would not work. RPC is outdated and the current 64 bit versions of Windows just did not want to work with our program to make RPC calls.
For this problem, we proposed the solution of running the FEP Simulator on a Linux operating system on the TMC network and communicating with it over the network.
We first thought to run the FEP Simulator on a Raspberry Pi, but later found it a better idea to run it on a Linux Virtual Machine on the TMC network.
Also, at first, we were writing traffic conditions to a file on the Linux Virtual Machine and having a cronjob run the FEP Simulator at a scheduled interval. We later changed the FEP Simulator to run persistently and await socket messages from the CAD Server, to avoid file system dependency and timing complications.
Although this adds an extra device to the system, our FEP Simulator actually resembles the FEP in that it is a both a separate hardware and software component in the system with clearly specified interfaces and protocols so it is easily maintained. In the future it would be possible to integrate it into the TMC Simulation software if a good Java RPC library becomes available.
The Green Dot
One of the biggest challenges in developing this system was setting up the FEP Sim to send a single fep_reply to the ATMS successfully. Initially, we didn't even know to send an fep_reply struct, we only knew to communicate with it via RPC. A single successful fep_reply is indicated by a "green dot" on the ATMS Client GUI. It took weeks of work setting up RPC correctly, reading FEP documentation and picking apart UCI plugin code to achieve this feat.
The Yellow and Red Dots
After successfully sending an fep_reply and achieving a "green dot", we needed to figure out how to make the dot yellow and red. We knew that the colors were a representation of two values, the volume and occupancy, but did not know what values they needed to be. We tried just about every combination of integers we could think of and all of them resulted in red dots with "call failed" status messages on the ATMS Client. It was later found that the occupancy value was supposed to be a float, not an integer, in the range [0, 1]. After this revelation, it was easy enough to find the values needed for yellow and red dots.
Getting the Full List of Traffic Network Data from ATMS
When we started the project, we had no traffic network data. We eventually found the initial "triangle" of traffic network data buried on the Paramics plugin PC. We made this data work, but it was not the full list of stations, loop detectors, and lines we had hoped for.
After some time poking around on the ATMS, we found a SQL dump file of all the data in the network. We wrote a program that configured this data into the format used by our traffic model, and it worked to successfully show data for all stations and loop detectors in the traffic network.
Creating the Highways Model
The ATMS and FEP only see the abstractions of FEPLines, which contain Stations, which contain Loop Detectors. As developers, we needed an abstraction of a highway. Naturally, we thought that a highway would have direction. For example, 5 South and 5 North. The problem here was we did not know the lane types, and some of the Loop Detectors in Stations actually had data for the opposing side of the highway. This made it extremely difficult to make a directed highway abstraction, so we decided to have an undirected highway abstraction. To apply traffic conditions, we simply specify which direction on the highway we want to apply a "dotcolor" and iterate through the stations and apply the color to lanes that are known to be in the specified direction.
Discovering Lane Types
Finding out what lane types represent is an ongoing battle. Although we now have a better understanding of what certain lane types are, there are still many unknowns.
This proved to be a very difficult problem. Lane types are represented by a "lane type id" and "lane description". These attributes are not 100% consistent in representing a specific lane type. As an example, an HOV lane can be represented as "SD HOV" or "HV HOV" and that is only for one direction. There is also an immense amount of lane types. Adding to the difficulty is that stations can have "ML" direction lanes and "OP" direction lanes for every lane time and it is not clear how this is represented for many lane types. Also, highways are not uniform by nature. In sum, the nonuniformity of highways, inconsistencies in lane type identification, and complexity of lane types has hindered us in fully understanding individual lane types.
Lane Control
Because lane types are hard to discover, having individual lane control has also been a challenge. If we cannot understand a lane type, we cannot control it programmatically so to speak. How can we reasonably control traffic at the lane level if we do not even understand what the lane types are fully?
The solution to this problem is the TrafficEventsEditor?. The TrafficEventsEditor? allows the user to create lane level batch scripts in a user-friendly GUI, that will be read by the TrafficModelManager? to simulate traffic conditions. The reason the TrafficEventsEditor? helps to solve the lane-level control problem is because the user can apply a traffic condition to a specific lane in the model, and then "preview" it on the ATMS Client. This way, if the user is unsure of the lane type, they can easily preview the TrafficLaneEvent? on the client and see if it has the desired results. This makes it easy to discover lane types. Another benefit of the TrafficEventsEditor? is the ability to quickly produce the TrafficLaneEvent? batch script that is read by the TrafficModelManager?, as opposed to hard coding the batch script.
ATMS Client Lag
For unknown reasons, the ATMS Client lags when the fep_replys are being sent to the ATMS. During the time the fep_replys are transmitted the ATMS Client appears to hang and will accept no user input. This issue needs to be investigated further and solved because it severely impacts the user experience.
Future Work
Here are our priorities for tasks to complete in the next phase of development.
- Solve the ATMS Client Lag problem.
- Implementing TrafficLaneEvents into the TrafficModelManager.
- Improving the TrafficEventsEditor as necessary based on user feedback.
- Enhance FEP Simulator to realistically combine traffic events from multiple incidents.
- Developing an agent that simulates traffic conditions as opposed to loading traffic conditions from a data file.
- Start the FEP Simulator automatically upon system startup and have it run in the background as a system service (daemon).
