Proinet
← Case studies
Manufacturing and trading (SME)

IT infrastructure audit — a degraded array and a backup on the same server

A remote audit of a Windows server running Hyper-V and SQL Server, ZFS storage, backups and the firewall of a manufacturing and trading company. We found a ZFS pool running without full redundancy and a disk predicted to fail within 24 hours, before any data was lost. The client received 9 recommendations, ordered the way they need to be carried out.

SMART failure prediction for one of the array's disks
< 24 h
power-on hours of the storage disks (10–15 years)
88–135k h
areas rated high risk
4 of 7
recommendations across three priorities
9

The challenge

The whole company ran on a handful of machines. A Windows Server 2019 host with Hyper-V ran SQL Server, the accounting system and a few supporting services. Next to it were a FreeNAS file server on ZFS and a pfSense firewall. The server had been in service for 7–10 years and the storage for even longer. Because everything worked, nobody had looked at the disks, the updates or the backups in a long time.

The client wanted to know two things: what actually threatens business continuity, and in what order to fix it.

How we worked

The audit was remote and diagnostic. The only change we made to production was cleaning up the VSS configuration (see below).

  • Windows server: OS configuration, event logs, services and processes, memory and page file usage, VSS / Shadow Copies, volumes and the array, update status, the Hyper-V environment, SQL Server and application services. We also checked the backups: what is copied, where it goes, and whether the system can actually be restored from it.
  • Storage (FreeNAS): ZFS pool configuration and health (zpool status, scrub results, checksum errors), SMART data for every disk (overall assessment, power-on hours), utilisation, redundancy and system version.
  • Network: the firewall platform and its pfSense version.

Every finding got a risk rating and a plain statement of what it means for running the business.

Findings: storage

  • ZFS pool DEGRADED. The RAIDZ2 array had lost one disk and was running with reduced redundancy. The good news is that ZFS had kept the data intact: the scrub finished clean and there were no checksum errors. The data was fine, but the safety margin had shrunk.
  • A disk in critical condition. One of the pool's disks reported a SMART result of FAILED, with failure expected within 24 hours. It needed replacing right away.
  • Worn-out disks. SMART showed 88,000–135,000 power-on hours, which is 10–15 years of continuous operation. The storage was long past its recommended service life, so after one disk is replaced, another failure is likely. Risk: high.
  • Too little space. About 231 GB in total, with roughly 40 GB free. A full image of the Windows server would not fit, so this box cannot serve as the backup repository.
  • Unsupported system. FreeNAS on FreeBSD 11.3, a 2020 build. It lacks security fixes, ships outdated SMB and OpenSSL components and is increasingly incompatible with newer software. All of this makes recovery harder when something goes wrong.

Findings: Windows server

  • Years of missed security updates. Unpatched RDP, SMB and Hyper-V vulnerabilities on the machine holding the database and the accounting system put the server at real risk of compromise. Risk: high.
  • Misconfigured VSS. Shadow copy storage had no limit (UNBOUNDED), there were far too many shadow copies, and the event log showed Volsnap / FilterManager errors. As a result, snapshots were eating disk space, and the snapshots and the backups built on them could become unreliable. We fixed this during the audit. VSS now has a 95 GB cap and a controlled snapshot schedule. Since the change it has been stable and has stopped consuming the disk.
  • Backup on the same machine. A separate disk in the server held a Windows Server system image. That is a real asset: it allows recovery from a logical failure or a move to new hardware. But it sits in the same chassis as the system it protects, with no offline copy and no external storage. If the host fails, the storage is damaged or ransomware strikes, production and backup are lost together. Risk: high.
  • 16 GB of RAM for everything. That memory was shared by SQL Server, Hyper-V, a Wi-Fi network controller with its own MongoDB and Java runtime, and Windows Defender. Memory use was high, the page file was busy and there was little headroom. Risk: medium.
  • Too many roles on one box. Network services ran next to the database and the accounting system. That means extra load, a larger attack surface and a harder migration later. We recommended gradually separating infrastructure services from the SQL and accounting environment.
  • No Active Directory. The server runs in a workgroup, with no central identity management. We recorded this as an input for future modernisation, not as something urgent.

Findings: firewall

The firewall hardware is still adequate for a small business, but pfSense CE 2.6.0 is behind on updates. We recommended moving to the current supported release. Risk: medium.

Remediation plan

The order follows one rule: first protect the data that is already at risk, then remove the causes, and only then improve performance.

Critical: now

  1. Extend the backups. Keep the local system image and add an external or offline copy, separate from the production host, with more capacity and regular restore tests. This comes first because every later step, from swapping a disk to applying updates, is only safe once a copy exists outside the host.
  2. Replace the failing disk in the ZFS pool and rebuild redundancy.
  3. Avoid restarting the storage until the backups are done, the disk is replaced and a migration plan is ready. Disks this old may not spin up again after a shutdown.

High: the coming months

  1. A new NAS for backups, large enough for a full server image, with a new backup strategy.
  2. Update Windows Server in stages, after a full backup, in a planned maintenance window. Years of missed patches on a database server are too much to apply in one go.
  3. Migrate the storage to a new platform running a current, supported TrueNAS.

Medium

  1. Increase RAM to at least 32 GB ECC.
  2. Update the BIOS and firmware of the server.
  3. Update pfSense on site, after backing up the configuration. If a firewall upgrade fails, remote access to the network goes with it.

At the hardware refresh: 12–24 months. The environment works, but it is at the end of its life cycle. We recommended a migration plan to a new server and new backup storage, with a proper backup and disaster recovery strategy. The work is spread over time, so not everything has to be replaced at once.

The result

  • The data risk was caught before a failure. We found the degraded array and the failing disk before another disk gave out. The data was still intact (clean scrub, zero checksum errors), so it could be protected instead of recovered.
  • An honest view of the backups. The client now knows what the current backup covers (a logical system failure, a migration) and what it does not (host failure, ransomware, storage damage).
  • One problem fixed on the spot. VSS now runs with a cap and a schedule instead of growing without limit.
  • A plan, not just a list of problems. Nine recommendations in three priorities and a 12–24 month modernisation path, each with the reason for its place in the order.