CDS Forecasting Framework

In addition to showing the new system, Nathan Bunker discussed the Open Immunization Software technical goals for the next 6 months as part of a road map to provide basic CDS software support. Many of these goals would be accomplished through a set of collaborative efforts with the CDS work group and its participants.

Collect & Organize Test Cases

The CDS work group has a sub work group that is working on the tedious task of creating a large set of test case records. Originally they planned to to collate test records from several sources but realized the process of synchronizing them all into one system was a daunting task and opted to create entirely new test cases. These test cases are currently being stored in an Excel spreadsheet. The goal of OIS to take this data and integrate it into a test system that will:

• Store and categorize all test cases, including the patient info, vaccination info, and expected results.
• Allow users to add comments and document test cases.
• Eventually support querying CDS engines and verifying if expected and actual results match.

Currently several project participants are interested in submitting their test cases to a central system. This effort will require Nathan to make some changes to the base software to accommodate each test set and then a representative from the submitting site to actually enter the data in. Nathan will provide support by making tools available to make the data transfer as easy as possible.

Estimated Effort (per entity)

• Nathan Bunker: 4 - 20 hours
• Entity representative: 10 - 80 hours

Improve Test Case System

The Texas Children's Hospital validation service is currently targeted for improvement to meet these goals. Most of the functionality needed is already supported by the TCH forecast validator, but it will need to be improved to support collaboration on the national scale. In addition the service will need to move to a more permanent location.

Estimated Effort

• Nathan Bunker: 40-80 hours

Improvements to be Made

• Create login and registration screen. Allows workgroup members to automatically register and login. Shows splash page with TCH legal language.
• Add concept of test sets. Every user will have a personal test set, and test cases exist in one or more test sets. Initially a user can create and upload any test cases and the group can decide which ones are promoted to a national test set.
• Add ability to better show and enter in expected results from many different registered users displayed agains the actual results from the targeted system. Current the software assumes the user is from TCH and is targeting improvements on the TCH forecaster. The user needs to be able to enter in expect results representing themselves or their organization and then compare against a chosen forecaster for verify actual results.
• Improve process for making comments so that it supports commentary from the community more effectively. Current there are comments but improvements could be made to them.
• Add support for relative dates in test cases.

Test Case Management and Verification System

The final system would ideally serve as a hub for all testing and verification work. It would capture all expert decisions and allow for forming permanent and ad-hoc testing sets. This system has the potential of becoming a national resource that the CDC could maintain in the years to come to support IISSB project goals. OIS wants to put the initial system in place to demonstrate the need and benefit of such a system.

Create Supporting CDS Tables

There are two sets of data tables that drive the CDS. The first is the basic data that includes vaccine codes, disease groups, and vaccine groups. This part is nearly complete but needs input from the CDS group as a whole to ensure that it is correct. The second set includes all the series table definitions that drive the decision logic. The CDS working group has made a lot of progress establishing these series and recording them in word document format. The next step is for Eric Larson and Nathan Bunker to meet and discuss the somewhat-final format for the decision tables, then for Nathan to put these into Google docs spreadsheet format. After this, Nathan will meet with each vaccine series representative and help them record the series correctly on the spreadsheets. After this is complete the word documents will not longer be used and the Google doc spreadsheets will become the official coordinated repository.

Estimated Effort

• Nathan Bunker: 10 - 15 hours

Implement Basic Infrastructure

There are still a few more changes to the basic state diagram, and after these are solidified Nathan Bunker will create a new CDS state diagram and update the support in the demonstration system. When this is complete the new system will be able to step through all the states but will not calculate all the required information. Most of the states will have no functionality.

Estimated Effort

• Nathan Bunker: 5 - 10 hours

Finish All State Logic

Once the infrastructure is in place, several technical members of the CDS work group can be assigned to flesh out the state blocks. The work should be highly modular at that point and perfect to involve everyone who wants to.

Estimated Effort

• Nathan Bunker: 5 - 30 hours
• Other developers: 10 - 80 hours

Develop Web Service Interface

Once the CDS demonstration system is working there will need to be an interface so that external validation systems can run tests against the demonstration system.

Estimated Effort

• Nathan Bunker: 20 - 30 hours

CDS Project Working Demonstration

The final system will essentially be a forecasting engine. How well it works will depend on how well the logic is implemented and how accurate the series data is.

Other CDS Engines...

The validation engine could also be integrated with other CDS engines. This would assist in the test development and serve as a way to allow these disparate projects see how well their systems are working compared to the national standards.

Estimated Effort (per web service engine)

• Nathan Bunker: 10 - 30 hours