|
CITY OF VANCOUVER
ADMINISTRATIVE REPORT
|
|
|
Date:
|
October 21, 2003
|
|
|
Author:
|
Martin Crocker
|
|
|
Phone No.:
|
7647
|
|
|
RTS No.:
|
03571
|
|
|
CC File No.:
|
1151
|
|
|
Meeting Date:
|
November 4, 2003
|
TO:
|
Vancouver City Council
|
FROM:
|
Director of Information Technology
|
SUBJECT:
|
Information Technology Infrastructure - 2003 Expansion and Replacement Program
|
RECOMMENDATIONS
GENERAL MANAGER'S COMMENTS
The General Manager of Corporate Services notes that the City's digitally-stored information is an extremely important asset that has been developed over decades and is critical to the timely delivery of City services. The recommendations in this report not only bring forward the annual portion of a longer term program to ensure the City's IT infrastructure is able to support the delivery of those services in the future but also represent best practices for managing this resource.
COUNCIL POLICY
On December 8, 1994, Council, while considering the "Next Steps to Better City Government" reports, which included the Information Systems Plan Report, directed:
On July 23, 1996, Council approved a "Long Term Financing Strategy - Information Technology Replacement Program", that recommended, amongst other things:
Since 1997, Council has approved four major "Information Technology Infrastructure Expansion and Replacement" programs.
SUMMARY
This report seeks approval for the replacement of obsolete core information technology infrastructure, primarily relating to information storage.
PURPOSE
The purpose of this report is to seek approval to continue the replacement and build-out of the City's information technology, communications and systems infrastructure.
Council approved the last IT Infrastructure Expansion and Replacement Program on May 28, 2002. Highlights of the achievements of that program are described in the next section of this report.
This 2003 program recommends that the City:
· Replace and upgrade data storage systems to accommodate growth, increase availability, and protect the City's information assets against accidental or catastrophic loss;
· Replace obsolete web and database computers that serve both public and staff information needs, and
· Upgrade the Vancouver Public Library (VPL) telephone switch.
BACKGROUND
Information Technology (IT) Infrastructure Replacement
The City's information assets and communications resources are managed and made accessible through a complex IT infrastructure that consists broadly of:
· "Front-end" devices - desktop and laptop computers, handheld devices like Personal Digital Assistants (PDAs), and telephones;
· "Back-end" systems - data storage, application servers and telephone switches;
· A communications network that connects the two and permits information retrieval by, and communication between, both staff and public.
These infrastructure components have limited lives, typically of 3-5 years (although telephone equipment lasts longer), after which they become either physically or functionally obsolete.
The City manages replacement of obsolete IT infrastructure through a longer term, planned program. Annual programs, based on need, bring these long term requirements to a manageable scale. In recent years this "IT Infrastructure Replacement and Upgrade Program" has focused on a major communications network upgrade (2000-2002), and a major desktop "front-end" refresh (2002-2004).
Typically, through advances in technology, obsolete equipment is replaced with equipment that has a higher capacity and increased functionality - hence the description "replacement and upgrade". This process also serves to accommodate the increasing demands that the City, like other comparable organizations, places on its IT infrastructure.
A submission to the 2003-2005 Capital Plan, received by Council on June 25, 2002, described an anticipated need, during that period, to:
· Replace many of its local data storage (LAN) systems, which are both technologically and physically obsolete, with current technology. Where improved data connectivity between City facilities permits, these currently-distributed systems will be consolidated and replaced with a more enterprise-oriented shared storage system;
· Take the first steps towards integrating its data and voice systems;
· Increase the robustness of its information systems, including deployment of redundant telecommunications connections, mirrored application and data servers, and potentially out-of-province data replication;
· Upgrade other core systems, like e-mail, both public and private web servers, and security infrastructure.
The 2003 program addresses parts of the first, third and fourth of these items.
The 2002 Program
The status of items approved in the 2002 program is summarized in the following table:
Item
|
Approved Budget
|
Costs to Date
|
Status
|
Networking and Telecommunications
|
$670,000
|
$290,000
|
All sites inter-connected to minimum standards. Some deployments of optical fibre for redundancy and high-bandwidth requirements are outstanding.
|
Police System Upgrades
|
$220,000
|
$250,000
|
Data access to police incident data improved, backbone network and voice-mail systems upgraded, secure communications with external police teams enabled, obsolete UPS replaced.
|
Security Infrastructure
|
$100,000
|
$110,000
|
Intrusion detection systems deployed, vulnerability testing software acquired and in use.
|
Desktop Computing Infrastructure Refresh (3-year program)
|
$12,050,000
|
$4,500,000
|
Approximately 1,700 desktop and laptop computers replaced (out of 4,000). Within budget and on schedule for 2004 completion.
|
DISCUSSION
The 2003 program presented in this report consists of three sub-programs:
|
|
|
|
|
|
|
Sub-Program
|
Description
|
Capital Cost
|
Annual Cost Increase
|
1.0
|
Refresh Data Storage Systems
|
Replace and upgrade data storage infrastructure to accommodate growth, increase availability, and protect the City's information assets against loss.
|
$915,000*
|
$25,000
|
2.0
|
Replace Web and Database Servers
|
Replace obsolete web and database servers that serve both public and staff information needs
|
$100,000
|
$5,000
|
3.0
|
Upgrade VPL Telephone System
|
Upgrade Vancouver Public Library telephone system hardware and software
|
$60,000
|
$0
|
|
|
$1,075,000
|
$30,000
|
* Includes only the first year of a two-year program.
A brief description of each of these programs, identifying the business drivers and presenting an overview of the proposed technology implementation and the anticipated benefits, follows. A more detailed description of the Data Storage Refresh sub-program is provided in the Appendix.
1.0 Data Storage Refresh (Two-year program)
The City's computerized data assets include:
· Office productivity files (documents, spreadsheets, small databases, presentations, and graphics files) for the more than 4,000 staff who use microcomputers;
· "Line-of-business" data directly associated with the City's delivery of services;
· Financial, human resource and inventory records stored in the SAP system, and
· E-mail correspondence and calendars.
The volume of primary on-line data, already at 1.7 TB (1 TerraByte is roughly 1,000,000,000,000 characters of data), is growing at a rate of more than 30% annually. This data is stored on over 150 computers, generally referred to as either file or application servers, depending on function.
There are challenges to managing this environment:
· Hardware and software obsolescence;
· Security vulnerabilities;
· Storage demand growth;
· Data backup and restore, and
· Potential interruptions of service.
To address these challenges, this report proposes that the City:
· Implement a consolidated data store and consolidate campus file servers;
· Replace off-campus file servers;
· Replace the current operating system software;
· Replace or upgrade the tape backup systems;
· Provision a secondary data centre at Chess Street, and
· Implement systems management tools and related IT processes.
This proposal replaces aging infrastructure and accommodates future growth. It protects the City's data resources and their availability through better management and by continuing to deploy secure, hardened data centres like E-Comm and the new back-up City facility at Chess Street, adjacent to the new Public Works yard.
The two-year program budget is summarized below. Additional operating costs result from net increases in the costs of annual hardware maintenance and software licensing.
|
Item
|
Capital Cost
|
Operating Cost increase
|
Implement consolidated data store, consolidate campus file servers, and replace the current operating system software
|
$450,000
|
$25,000
|
Replace off-campus file servers and the current operating system software
|
$635,000
|
$18,000
|
Replace/upgrade tape libraries
|
$370,000
|
$4,000
|
Provision Chess Street data centre
|
$380,000
|
$43,000
|
Implement system management & monitoring
|
$125,000
|
$10,000
|
Project management & training
|
$190,000
|
|
|
|
$2,150,000
|
$100,000
|
The program will be funded over two years, with the most pressing needs addressed first (in particular, the new tape library needs to be in place by year-end, 2003). The balance of the program, subject to funding approval, will continue through much of 2004. The cost breakdown between the two years is:
|
Year
|
Capital Cost
|
Associated additional operating costs
|
2003
|
$915,000
|
$25,000
|
2004
|
$1,235,000
|
$75,000
|
|
TOTAL
|
$2,150,000
|
$100,000
|
This report requests approval of a program totalling $2.15 million, however, completing this program is subject to approval of the 2004 Basic Capital Budget as discussed in the Financial Implications section.
2.0 Replace Web and Database Servers (One-year program)
2.1 Web Servers ($60,000)
The City delivers information and services to the public and staff through two separate web servers. Smaller servers are used for application development and testing.
The server that supports the public web site provides 30,000 pages of information on Vancouver, its government, services, communities and history. Over 15,000 visitors visit the site on a daily basis, viewing 80,000 pages of information. This server is over 5 years old, and has seen enormous growth in use over that time. It is running at capacity and, at peak use periods, can no longer deliver acceptable response. It must be replaced.
The staff site does not experience the same load (5,000 pages of information, 2,500 visitors daily), but it is also 5 years old. Many internal processes are dependent on its availability, and it too should be replaced.
Both servers are running old web-serving software that is no longer supported by the manufacturer, and it is time to replace it. For operating system consistency with the rest of the server fleet (see discussion in appendix), the recommendation is to migrate to Microsoft's Internet Information Server (IIS) on the Windows 2003 platform.
Other components planned for deployment are a separate application server, which will balance the load between information pages and service delivery, and a combined development, test and quality assurance server which will replace two desktop-class microcomputers currently serving those roles.
2.2 Database Server ($40,000)
Many of the applications that currently deliver services to the public and support internal processes through the City's two web servers retrieve information from a database server. This server supports over 70 databases that are core to the City's operations, including community event postings, Park Board recreation guides, and the QuickFind staff and services directory. It represents a significant potential point of failure. The database serving software also needs to be upgraded.
It is proposed to acquire a new primary database server to be located at the E-Comm data centre, relocating the existing server, which is almost 4 years old but still capable of functioning in a back-up role, to the Chess Street site. Each will use the shared storage facilities at that site, and replication software will synchronize the databases at the secondary site with those at the primary site. If the primary server fails, it will be possible to bring up the secondary server within a short period of time.
Additional operating costs of $5,000 result from the licensing and maintenance of the Chess Street server.
3.0 Upgrade VPL Phone System (One-year program)
The Mitel phone system used to provide all VPL phone and voice-mail services at Library Square was installed in 1995. Vendor support for both the hardware and the software ceased about 2 years ago, creating a risk of a major disruption of services in the event of any failure.
This upgrade provides a limited capacity to deploy Voice-over-IP (VoIP) phones that communicate over the data network rather than over dedicated phone lines - an example of the move towards integration of voice and data systems identified in the 2003-2005 capital plan submission.
Cost of the upgrade is $60,000.
Funding for the initiatives described in this report will flow through the Information Technology Long Term Financing Plan. This plan is funded from a transfer from the annual Operating Budget and provides a mechanism to ensure that a constant annual source of funding can be balanced with the highly variable pattern of expenditures that characterizes the IT replacement program.
This report requests approval of program expenditures totalling $2.31 million. Of this amount, $1,075,000 is requested in 2003:
· $175,000 of the funding approved for replacement and upgrade of the LAN infrastructure in the 2002 IT Infrastructure program is unspent (in anticipation of this program, non-essential replacements and upgrades have been deferred) and can be applied to the Data Storage Refresh sub-program;
· The balance of $900,000 was approved in the 2003 Basic Capital Budget.
|
|
|
Sources of Funds
|
|
|
|
|
|
|
|
Carried over from 2002 IT Infrastructure program
|
$175,000
|
|
|
2003 Basic Capital Budget
|
$900,000
|
|
|
|
|
|
|
Total
|
$1,075,000
|
The balance of the funding for the Data Storage Refresh Program ($1,235,000) will be available subject to Council's approval of the 2004 Capital Budget in early 2004. Work on these components of the program will await approval of that funding.
CONCLUSION
The City is increasingly dependent on its electronically-stored information and associated delivery systems. These systems demand a technology infrastructure that extends to all parts of the organization, is fast, dependable and robust. The City must also protect itself and the public it represents against loss of data and interruptions in service, both of which risks can be mitigated. Finally, it has a duty to make reasonable efforts to ensure that in the event of a disaster, it is equipped with the information and communication resources that would be needed to quickly adopt the leadership and supportiveness that its public would expect of it.
The recommendations of this report attempt to address these goals.
- - - - -
APPENDIX
DATA STORAGE REFRESH PROGRAM DETAIL
As discussed briefly in the body of this report, the City manages extensive data resources:
· The total volume of primary on-line data is 1.7 TB;
· The composite growth rate is in excess of 30% annually, attributable to ongoing data capture, an increasing volume of history, and new forms of information, like digital photographs, that are particularly storage-intensive;
· Within 5 years, on-line data storage requirements are projected to be over 6 TB;
· The requirements for secondary magnetic tape-based storage, which provides protection against loss, statutory records retention and archival storage, are 8 or 9 times greater;
· The primary data is stored on over 150 computers, generally referred to as either file or application servers, depending on function.
There are challenges to managing this environment:
Obsolete server hardware
The majority of the file servers are over 4 years old and must be considered obsolete. As this kind of equipment ages, frequency of hardware failures increases, and replacement or expansion parts are difficult or impossible to obtain.
Obsolete operating software
Operating system software requires maintenance, typically out-of-hours as it necessitates a service interruption. Because of the number of servers, the diversity of operating systems, the absence of a systems management framework and the lack of appropriate tools, it has not been possible to maintain this software consistently. As a result, the majority of operating systems on City servers are two or three generations old, and manufacturer support has expired for some.
Security vulnerabilities
With the growing prevalence of computer viruses and other malicious software, it is important to protect systems by maintaining their operating software at current release levels and applying security patches as soon as they become available. The City's inability to do this in a timely manner puts City data and systems at risk. Several of these deficiencies have been noted in both external and internal audit reports.
Storage growth
Each individual server must either be initially over-provisioned with storage (at significant cost) or storage must be periodically upgraded to accommodate growth. Upgrading storage is a labour-intensive process that necessitates the server being unavailable for hours or even days, and has to be performed outside normal business hours. The City spends an estimated $200,000 annually on storage upgrades, including staff overtime costs.
Data backup and restore
The data on each server is backed up nightly across the network to a centralized tape storage device, known as a tape library. Data volumes are such that the library is at capacity, parts of the network are approaching capacity, and centralized backups of some servers are not possible.
In the event of a single storage failure, restoration of data from magnetic tape and resumption of service can take as long as 24 hours for a large server.
Interruptions of service
A variety of scenarios, ranging from scheduled maintenance to failure of a single component to catastrophic loss of a data facility, can cause interruptions in service. The impact may be minimal - technical staff working through the night or over a weekend, with only overtime costs and minor inconvenience to other staff and the public. The potential exists, however, for a major failure to interrupt service for days or even weeks. During this time, staff who depend on the affected systems would be unable to deliver their mandated services, at the cost of lost productivity and potentially major inconvenience to the public. While the City has moved its key servers to the earthquake-hardened and secure E-Comm facility, these servers are potential points of failure in the City's business continuity strategy.
As much of the storage infrastructure is obsolete and scheduled for replacement, the City has an opportunity to establish a more secure foundation for management of its data. Current technologies permit (and best practices encourage) the City to address the challenges identified above through the following actions:
Implement a consolidated data store and consolidate campus file servers ($450,000):
· A data storage device, known as a Storage Area Network (SAN), will be deployed to support the data requirements of the key servers at the E-Comm data centre. It will offer improved data protection, easier storage expansion, reduced administrative complexity, and specialized data management functionality. Starting at 4 TB capacity, funding is included for a further 4 TB to meet the City's projected 5-year data storage requirements. A Request for Proposals will be issued in Q4, 2003.
· 11 servers provide staff on the main City Hall campus and some closely-connected locations (served by optical fibre telecommunications) with access to office productivity files. These will be consolidated to a pair of servers, taking advantage of both the increasing power of current technology and the shared storage device described above. The primary benefit of this consolidation is the significant reduction in maintenance workload. In turn this pair of servers will be "clustered", meaning that they both share the workload, but if either one fails the other can continue to provide service, though possibly at some performance impact. A secondary benefit is the potential to perform maintenance on one while the other carries the workload, reducing the need to perform this maintenance out of hours.
Replace off-campus file servers ($635,000):
· These are servers (mostly 4 - 8 years old and well past their anticipated lifespan) that store local information for community centres, recreation facilities and other sites that have insufficient telecommunications bandwidth to share centralized storage. While centralization and consolidation are the goals, in some cases a local file server with its own storage is still the best approach. In these cases the server must simply be replaced with current hardware.
Included in this category are 6 VPL servers that serve staff, public, e-mail, marketing, Help Desk and backup functions. VPL technical staff are analyzing their requirements and expect to have a replacement approach developed by year-end 2003, for implementation early in 2004. Participation in the strategy described in this report is one option, but VPL has unique requirements that may best be satisfied through a different or hybrid approach.
Replace the current operating system software (costs included in items above):
· The City adopted the Novell Netware file server operating system in 1995, when it was the de facto standard. It has served the City well for eight years but is now approaching the end of its life;
· The proposed replacement is Microsoft's Windows 2003, the latest release of the server operating system family which has captured a large share of the market. As this is also the operating system used for application servers in the City, it will become the standard for at least 90% of the City's server fleet. This homogeneity considerably simplifies the management task, facilitating timely maintenance and offering the best protection against security threats;
· With the replacement of both hardware and software, the City's store of documents must be migrated from the old environment to the new. Some synergy exists between this project and:
o A Records Inventory and Classification project being pursued concurrently, whose goal is to better manage (organize, store and ultimately dispose of) both paper and electronics records holdings, and
o An audit recommendation that information access controls be reviewed to ensure that confidential information is accessible only to staff with a need to know.
The Records Inventory and Classification project will analyze existing storage of electronic documents to identify how records series are stored by directory, and may recommend a new storage scheme based on classification and security requirements and information management best practices.
This (Data Storage Refresh) project will provide for local administrator training, physical migration of the files, and their reorganization consistent with the recommendations from the Records Inventory and Classification project. It will also provide temporary data storage that may be needed during re-organization.
Replace or upgrade the tape libraries ($370,000):
· The smaller library, now 5 years old, will be replaced with a new library with 50 TB of storage capacity. It will be relocated from City Hall to the E-Comm data centre;
· The other library, now 3 years old, will be either upgraded or replaced and relocated at Chess Street;
· The benefit of this approach is that both the E-Comm and the Chess Street sites will be fully self-contained. Not only will they have copies of current data and servers capable of delivering it, they will also have access to full back-up and historical data should the current data be corrupted;
· Replacement of the tape libraries is a pressing need, so a Request for Proposals was issued in August and closed in September. Evaluation of the responses is proceeding, with award of a contract subject to approval of funding and anticipated deployment in late November, 2003.
Provision a secondary data centre at Chess Street ($380,000):
· A computer room was designed into the new Fire Training Centre at Chess Street. Adjacent to Firehall #1 and the new National Works Yard, and connected by optical fibre to E-Comm and City Hall, this is an ideal location for a stand-by data centre;
· The Chess Street data centre will house a secondary set of servers and data storage replicated from the primary E-Comm site, providing the ability to resume service should either an individual primary server fail or the whole primary site be lost (including a communications failure). In the interests of economy, this fail-over will not be seamless. There will typically be a short service interruption, and when service resumes, the level of service (like response time, number of concurrent users supported) may be impacted. Equipment at the secondary site will generally be less capable of sustaining a full production workload. However, the resumption of service with little or no loss of data within minutes or (in a worst case) a few hours means that staff will continue to be able to provide service with a much shorter interruption than would now be imposed;
· Associated costs include provisioning the Chess Street data centre with an un-interruptible power supply (UPS) and telecommunications infrastructure, acquisition of secondary storage and two stand-by servers, and replication software.
Implement systems management tools and related IT processes ($125,000):
· The tools will monitor the City's server "fleet" and provide prompt warning in the event of impending or actual failure of any component, will improve security through centralized administration and event logging, will facilitate operating system software maintenance, and will help to ensure that the service levels expected and needed by the user community are met;
· The processes are a recognition that management of data storage, maintenance of the server fleet, and continuing refinement and testing of the business continuity strategy are sustainment disciplines more than project tasks. Some internal work re-assignments will be made to ensure that the resources are sufficient to carry out these responsibilities.
Project management and training ($190,000):
This program involves a significant commitment of both financial and staff resources. The staff resources needed are not available entirely within the City, and the project budget includes funding for project management, contract and consulting resources. It also includes funding for training in the new file server operating system, primarily for the departmental system administrators.
* * * * *
