Municipality of Trento

The city of Trento is located in the north-east of Italy and has about 100.000 inhabitants. The ‘Sistema Informativo’ department delivers most part of the Information Technology services to the municipality. The IT infrastructure counts more than 1.200 workstations which are distributed among about 20 different locations and connected via a city-owned backbone (optical fiber network) and several satellite WANs.

Over 30 employees work for Sistema Informativo managing the complete infrastructure. Their main tasks consist of on-site hardware/software support and maintenance, software development of vertical applications, System and Network administration. All that persons have many years of experience in their respective fields; many of them formerly worked in the private sector, or at the local University.

“What we see as the main problem with proprietary software, even if it's feature set is complete, is that you don't have things firmly under control. You have neither the chance to drill down to track problems, nor to ask someone you trust to do this on your behalf. You cannot decide when to update or upgrade, solely basing on your needs, because it's the software license owner who decides the timings when he wants and basing on his needs. In case you need profound customizations you can't do them, without asking the license owner. This whole situation with proprietary software has very strong impacts on Public Administrations, because they have to be particularly independent, especially in IT domain.

FLOSS Software to grant Citizens Access to Public Services

“The mission of public administration is to: ‘Serve the citizens as best as I can’. ‘Best’ means that the citizen's data has to be accessible, forever, and without any constraints; data has to be safe and protected from unauthorized access. These requirements of Public Administration services mean that they are best built on Open Standards and Technologies, allowing citizens to access them, for instance, with their Operating System of choice. FLOSS Software is the only way to grant all of these demands.

“In Italy a law states the ‘digital rights’ for citizens in dealing with Public Administration called the ‘Law for Digital Administration’ ("Codice per l'Amministrazione Digitale"); article 68 clearly assigns a strong preference to FLOSS Software. Anyway, the freedom Free Software deserves does not come for free.

Two Strategies for Choosing a Suitable Software Solution

“To choose the suitable solutions, a strong competence is needed, and many times one single FLOSS solution is not suiting best your needs, but only a combination of some of them (Note: The same is applicable in general for proprietary solutions as well).

“In many cases the software feature set is lacking something fundamental you need for your scenario. So, in general you could say that instead of investing in a large feature set (most of which is not valuable for you because you actually won't need it) in terms of proprietary license cost, you shift the investment towards tailoring individual features of FLOSS software on your needs. This requires a strategically move in one (or better: both) of the following two directions:

1. You have to buy expertise from an external person or company you trust.
2. You have to leverage more and more on your internal expertise.

“So a FLOSS solution is not necessarily less expensive than a proprietary one; but the key argument is that you can choose on HOW and on WHAT to spend your money. And in general that money does not nourish yet another global player, but can be regarded as an opportunity for local economy (choice 1) or a way for increasing internal team value (choice 2). FLOSS is then the best way to increase the value of a well-motivated team, if it happens you have one. It turns what is ordinarily only regarded as "labor cost" into a productive investment in ‘human resources’.

Combining Internal with External Expertise

“In our experience, the best results come from a combination of the two approaches, because relying only in internal expertise could lead to ‘blind alleys’ where technical solutions are over-engineered and difficult to maintain in the long term. A partnership with (carefully chosen) external expertise may lead to a real community, where ideas and solutions are freely discussed and becomes more easily exportable to other public bodies (which are a real must for cutting costs in Public Administration as a whole).

How Proxmox VE fits into this Strategy

“Proxmox VE is a real use case for these concepts; we heard of it for the first time some years ago, attending a sysadmin course organized by the local Linux User Group. It was PVE version 1.4, if I remember well, and the person talking, Giuliano ‘Diaolin’ Natali, is one of the prominent FLOSS experts and Entrepreneurs in our province (his company, OpenIt, is now Proxmox partner).

“So we came over Proxmox VE and found it to be an ideal virtualization solution. It is built on Debian GNU/Linux, which was already our distribution of choice for Linux servers, so it was easy to integrate and we could benefit from the already existing know-how of our team.

“In addition, Proxmox is based on KVM, the most promising free/libre software solution for hardware virtualization. But it also offers OpenVZ as a lightweight container based alternative. To help us simplify management, it provides a very nice and powerful web based interface, out of the box. Additionally, Proxmox Server Solutions, the company behind the project, is offering scalable support options; as our needs grow, we can easily scale which gives us a lot of flexibility.

“Our server hardware was gradually being phased out, in favor of blade systems, which featured hardware virtualization. This enabled us to afford KVM virtualization for all our servers (formerly only Linux ones were virtual, because only Linux allowed non-hardware virtualization). KVM was already part of the vanilla kernel, ensuring us not getting stuck in a proprietary solution.

Hardware Replacement Rate sets Data Center Consolidation

“The consolidation of the data center evolved naturally over one or two years, following hardware replacement rate. The entire "Sistema Informativo" data center is now built on the Proxmox VE platform. Currently, the department runs ten production instances of Proxmox VE, as well as three clusters that form the real core of the data center. In total, they have about 80 VMs, running a mixture of Microsoft Windows and Debian operating systems. With this setup they serve the needs of more than 1.200 internal workstations, and several on-line services. Most of the hosts are on blade hardware, served by a fiber channel Storage Area Network.

“Currently this infrastructure is managed by three system administrators, each one being involved in many other activities, not virtualization related, such as software development, user assistance, etc. The availability of the data center is now very high, with less than ten issues per year (and only one or two impacting end users).
“An example of how we leveraged on FLOSS flexibility in our PVE usage is the backup strategy.

Leveraging FLOSS Flexibility: Custom File System Backup Solution

Since reliable file system backup was a major issue, "Sistema Informativo" implemented a custom backup solution based on BackupPC and LVM based snapshots. Resoli explains:

“One of the immediate benefits we saw with Proxmox VE was the accomplished cost savings obtained replacing a very pricey proprietary snapshot feature of the SAN with host LVM based snapshots. With the LVM based snapshot feature provided by Linux Operating System, which is at the base of Proxmox VE, and thanks to the very smooth, modular and noninvasive integration of PVE features into the OS, we were able to build a custom backup solution based on BackupPC that exactly fits our needs. We contributed the solution to the Proxmox community where it is now available also to other users. This solution is now leveraging on Proxmox VE also on the storage side, combined with a custom offsite encrypting synchronization feature based on DRBD.
(see: Filesystem Level Backups with LVM Snapshots)

“The hardware setup comprises two twin servers with autonomous storage (in order the backup not to depend from SAN infrastructure), both with PVE onboard; one server is placed locally, and the other offsite. At the moment two BackupPC virtual machines are running on the local server, each one dealing with a 2TB backup pool.

“The storage is configured on the PVE physical host on three layers: LVM -> DRBD -> dmcrypt, the latter being presented to the vm. The DRBD layer is asynchronously connected with the remote PVE server. So, after nightly backups, the activation of the connection between the two DRBD peers is scheduled. Given that DRBD is under dm-crypt layer, all exchanged synchronization data are already encrypted, and remote data are encrypted as well. The local server performs one time a week a non-encrypted tape dump (using dump/restore standard unix commands) using a snapshot of the LVM pool volume and drbd - crypt layers created on the fly over it.

Benefits

• Generally speaking Proxmox VE allowed us to build a solid virtualization platform that fits exactly our needs;
• It is lightweight and easy to access thanks to an excellent web user interface.
• It allowed us to increase the availability of our services thanks to the live migration feature during updates/upgrades.
• Proxmox VE is highly customizable and easy to adapt to the evolving structure of our hardware setup.
• Last but not least, savings in license costs were geared in useful directions: buying support from Proxmox (really excellent), and acquiring expertise (internal or external).
• Consolidating all our servers on Proxmox VE reduced considerably the system administration burden, freeing precious resources to dedicate to our core business: Serve the Citizens.

“Over the years we have watched the Proxmox VE project flourish, strictly following the fast pace of development of KVM and OpenVZ features, but also integrating many other emerging open source technologies like GlusterFS or Ceph in a very nice fashion.

“Our main goals were to improve security and reliability and at the same time minimize the dependency on proprietary solutions. In conclusion, we found Proxmox VE a very effective, scalable, flexible and powerful virtualization solution, with constantly increasing features. Keeping in mind that it is a really open, clean and modular product, we are confident that Proxmox VE will satisfy our future needs, as well.”

Roberto Resoli
System Administrator and Senior Programmer, Sistema Informativo