Title: Summary of Discussion with Sylvia Davis & Neal Wormsley Subject: SAP / IRIS and Impacts on Research Meeting Date: January 25, 2002 Author: J. Douglas Birdwell, Research Council The following is an edited version of e-mail that I sent to Sylvia Davis and Neal Wormsley following our meeting on January 25, 2002. I requested comments & corrections from both, and I have incorporated those comments that I received. ================================================================ URL for discussion group: I have created a discussion group for issues related to the newly formed Strategic Planning Group on Computing and Networking. It is located at http://www.lit.net/; look for the item on the SPG in the sidebar to access the discussion forum. (You can also go directly to it at http://www.lit.net/cgi-bin/netforum/spgcn/a/1 .) We need to establish an ongoing and broad-based process whereby feedback can be discussed and acted upon, essentially as is done in industrial quality control, and I think the discussion forum format fits the information gathering function well. This can be an excellent method of obtaining information about problems users have with SAP/IRIS (or other systems & processes) and allowing anyone with an interest to participate in the discussion. It can provide feedback to both the committee that Neal Wormsley is forming to address SAP issues and to Neal's development & maintenance team members, who have a longer-term role. An invitation was supposed to go out within a few days of my initial e-mail to everyone on campus that asks for anyone interested to participate in the discussions on the forum. (Note that the invitation is on hold pending a meeting scheduled with the Provost's office, to be held in March.) Contract/Grant Reporting: I showed you an example of the type of running cost & cumulative cost report, which is an extremely common requirement imposed by external sponsors, and which I have not found a way to generate using SAP/IRIS. I didn't ever find a way to generate it with the old system, either, but that system predated this type of requirement. These reports have necessitated the creation, probably by most principle investigators, of "shadow" accounting systems -- what I referred to tongue-in-cheek as "keeping a double set of books", and require a significant -- in some cases major -- time commitment by PIs. This issue was not resolved at our meeting; it appears that at present SAP/IRIS can not generate this type of report. I think it would be useful for some of Neal's people to meet with PIs and examine the various systems and reporting methods that have been created to get around the deficiencies in our old system (and SAP). I suspect a good method would be to look at the shadow accounting & reporting practices of several completed projects by different PIs. I am willing to volunteer the systems I used for the past several years as one example. From these, perhaps a design for extensions to SAP/IRIS can be created that, when implemented, can reduce the need for the shadow accounting systems. Security: [Note: Sylvia Davis opted to let Neal Wormsley respond to these comments and issues; no response was received from Neal Wormsley.] We discussed SAP/IRIS security issues. The "post" vulnerability that appears to allow anyone with SAP/IRIS access to expend funds from an account without going through a chain of approvals is, according to Neal Wormsley, due to be fixed with the next release of SAP. The remaining question is when this release will occur, since until that time, the system is vulnerable to this exploit. A remaining issue is that SAP/IRIS does not appear to support access control lists, limiting access to users associated with an account. During the meeting I asserted that UT's SAP/IRIS systems have been compromised in recent intrusion attempts originating on the Internet, and I promised you supporting evidence. I am forwarding as a separate e-mail an e-mail from Rod Meryweather dated August 13, 2001 stating that several UTK systems including SAPITS1.ADMIN.UTK.EDU, SAPITS2.ADMIN.UTK.EDU, SAMITS3.ADMIN.UTK.EDU, SAPITS6.ADMIN.UTK.EDU, and SAPFAX.ADMIN.UTK.EDU were verified, using two separate scanners, to have been infected with the Code Red Worm II and to have active back doors. Note that the only reliable method of cleaning a system infected with the Code Red worm is to wipe the disk drives, reformat, and reinstall everything. If this method was not used, then the worm may remain. There is a documented case, for example, of a secretary's machine in my department whose system was infected when a client application was downloaded from a UTK administration server and installed. The secretary's machine's filesystem had to be wiped, and the operating system and applications reinstalled, because of inadequate security on the administration server that had allowed the worm to corrupt its files in the first place. I checked these systems today, and got responses from SAPITS1 and SAPITS5 indicating that SAPITS1 is running what appears to be some version of SAP, and that SAPITS5 is running the production Web user interface to SAP. The Code Red back door allows unrestricted access to all information stored on an infected machine, including accounting information. Thus, any systems that trust any one of these systems could also be compromised. I don't know how SAP maintains its internal database, but I'm assuming it is running on separate servers that are queried by software running under the web server front-ends. Since these web servers must have write/modify access to the databases in order to work, an infection of the web server machines can compromise the SAP database. Under a separate e-mail from Rod Meryweather dated September 21, 2001, which I am also forwarding, a listing of the status of Nimda infected systems is given. WWW.UTENN.EDU (128.169.188.72) is on this list with the notation: "(SAP, IMMS, FTP) Up, figuring out best way to take system down with out [sic] taking down payrole [sic]." I didn't explore this system further. During our conversation, I stated that I have firewall / server logs indicating that SAP machines have attacked my network. I was mistaken in this; I was thinking of other machines, and have not looked specifically for attacks by SAP machines. However, at this point this seems to be a moot point, as the campus information security officer has already identified SAP machines that have been compromised. A firewall can serve a variety of functions in the SAP environment: It can cut down on the number of network attacks that get through to the machines and protect ports that are not used by the SAP software and its web-based front-end. It can provide extensive auditing of network traffic with any of the servers. It can run intrusion detection software that automatically flags suspicious activity and denies further attempts from the offending IP addresses. Finally, it can reduce the opportunities for transmission of information out of a compromised system to a remote site. Although Neal indicated that he had been provided information that a firewall would have limited the functionality of the SAP system and accessibility from remote regions, this is in practice rarely the case. When it is the case, it is usually indicative of poorly designed server software -- as in the case of Banner -- rather than limitations of the firewall. The decision to implement a firewall to protect the SAP hardware should have been made at the beginning of the project. Since it wasn't, it should be done now. Similar issues arise with other campus resources, an example being all systems that maintain student records. SAP resources do need firewall protection, and the sooner it can be implemented, the better. Two other recommendations I will make: SAP access should be restricted to on campus only or through a virtual private network (VPN) from remote sites, which can still operate across the Internet. Second: ALL accesses to these servers should utilize encrypted protocols (SSL for web-based access, and SSH/SCP for everything else.) ECR process: There have been lots of rumors about the effort certification report (ECR) process, and our meeting cleared up some of those. Among them: (1) Although ECRs are to be filled out within a month, there is a six month sliding window during which they can be changed. (2) There is also a process in place to go back and fix problems after the six month window has passed, which requires human review and approval. (3) There are no limits on other (non-personnel) charges. An issue we did not discuss at the meeting is training of all personnel who must participate in the ECR processes. I don't know of any faculty who know how to use the system, yet. Ron Maples spoke at yesterday's Research Council meeting, and told us that self-effort certification is to begin in February, covering January activities. There is a very serious education problem here; I, quite frankly, don't see how it is possible to begin using these procedures in February -- next week -- without first mounting a significant training effort. Since it is essentially possible to ignore the ECR process, with the consequence that contacts and grants won't be billed, my suspicion is a lot of this will happen until a significant education effort is mounted. [Note: In response to my email with this summary, Sylvia told me that on-line effort certification will most likely not be ready until March. She also believes the system will require minimal time to train. In addition she noted that billings will still occur even though effort has not been approved in the system. Adjustments would then be generated later as required.] Another security problem -- this one, in my opinion, quite severe, was uncovered in yesterday's Research Council meeting during discussions with Ron Maples. The effort certification reporting process that is about to go into effect relies upon UT network identifiers and passwords of faculty and others who must certify effort. To date, this netid and password combination has been used primarily to update the campus ph (people search on the UTK web site) system, and security has not been terribly important. In particular, nothing in the system requires users to change their original password, which is automatically generated as a combination of birthday & month, and SSN, and thus is fairly trivial to crack, especially over a large population of users. Before the ECR process goes online, software should be put in place that REQUIRES all users to change from their default passwords, and CHECKS passwords to ensure that they meet adequate security guidelines. I've never used my netid/password for anything other than ph, so I'm one of the users that has never bothered to modify the default password (which I usually can't remember because it's almost never used). I don't know if this netid/password combination is used for other purposes or not, but if it is, there may already be significant security vulnerabilities. This is an example of what I meant when I made the comment that until very recently our university has had very little regard for acceptable computer & network security practices. Encumbrances: An encumbrance process will be in place before the end of this fiscal year. Encumbrances will be limited to the end of the current fiscal year or the anniversary date of a grant or contract, whichever is sooner. For multi-year awards, a funds reservation system (which already exists) will have to be used. SAP/IRIS versus Finance or other problems: Efforts need to be made to separate, or isolate, problems that people are experiencing by their cause. Delays in contract billing, for example, are probably due to finance department and research office problems rather than SAP/IRIS. The Finance department is assisting the Research Office in getting information into the system by providing a portion of a student work for their use. Direct costs & F&A: Direct cost charges can be separated from F&A (indirect) charges now; enhancements are still being made. Endowments: I don't really understand this category of problems since I don't deal with these accounts. My understanding is that your perspective is that the process in now "ok, but confusing". Human Resources: HR is on the new system now, so the problems that were seen last year should be reduced.