fail2ban time offset issues

While trying to set up fail2ban, I found that even though my regex/logs matched up nothing was being banned/caught by fail2ban

After a bit of investigation it seems that the auth.log time was being written in GMT whereas fail2ban was expecting it in BST:

==> /var/log/auth.log <==
Oct 11 20:52:21 ns2 sshd[18119]: Invalid user test from 1.2.3.4
Oct 11 20:52:21 ns2 sshd[18119]: Failed none for invalid user test from 1.2.3.4 port 47862 ssh2
Oct 11 20:52:28 ns2 sshd[18119]: Failed password for invalid user test from 1.2.3.4 port 47862 ssh2
==> /var/log/fail2ban.log <==
2010-10-11 21:52:04,017 fail2ban.filter: DEBUG  /var/log/auth.log has been modified
2010-10-11 21:52:04,029 fail2ban.filter.datedetector: DEBUG  Sorting the template list

Fairly simple fix of:

rm /etc/localtime
ln -s /usr/share/zoneinfo/Europe/London /etc/localtime

and I am now successfully banning myself from accessing my server.

MessageLabs Mail Filtering and Vague Errors

450 Requested action aborted [7.2] 20412, please visit www.messagelabs.com/support for more details about this error message.

It took a remarkably large amount of searching to find out what ‘[7.2]’ meant in this error message, and why we kept getting a mailserver’s IP blacklisted, but if this happens to you, hopefully this will help resolve it.

When MessageLabs returns a [7.2], this seems to mean that they’ve checked the IP address of the host which is connecting to their MX against the CBL. Connections will be dropped immediately, rather than mail being rejected, as such:

# telnet cluster8a.eu.messagelabs.com 25
Trying 85.158.143.51…
Connected to cluster8a.eu.messagelabs.com (85.158.143.51).
Escape character is ‘^]’.
450 Requested action aborted [7.2] 20412, please visit www.messagelabs.com/support for more details about this error message.
Connection closed by foreign host.

The easiest way to get around this is to fix your mail server, then request delisting from the CBL.

In a completely unrelated note (ahem), it seems that you may be added to the CBL if you send an email from a domain where the sending mail server is explicitly disallowed by SPF records (such as -all with no matching include), to a gmail address; Google will automatically (?) submit the IP address to the CBL and your problems will begin (again).

I highly recommend robtex as a lazy way to check your hosts against blacklists.

VMWare ESX and a full SQL Server Database

Hypothetical situation. You installed VMWare ESX, possibly upgraded from 3.5 to 4, went with the embedded SQL Server, and Many Years Later the VirtualCenter server no longer starts. You look through the event logs and the best you can find is:

Faulting application vpxd.exe, version 4.0.10021.0, faulting module kernel32.dll, version 5.2.3790.4480, fault address 0x0000bef7.

So you decide to look at general application eventlog events rather than just for VMware:

Could not allocate space for object ‘dbo.VPX_EVENT’.’PK_VPX_EVENT’ in database ‘VIM_VCDB’ because the ‘PRIMARY’ filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.

“Great”, you think. I can just pass this over to a DBA to get them to increase the filegroup size. Then you dig a bit deeper and look at the event log for SQLServer:

CREATE DATABASE or ALTER DATABASE failed because the resulting cumulative database size would exceed your licensed limit of 4096 MB per database.

“Oh no!” you sob. You really don’t want to try migrating to an enterprise database right now. Worry not, there’s a VMWare solution. This easy process is:

  • Install Microsoft SQL Server Management Studio Express
  • Download and extract VCDB_PURGE_MSSQL.zip
  • Make sure all VMWare VirtualCenter processes are stopped
  • Open Microsoft SQL Server Management Studio Express
  • File -> Open -> Choose the extracted sql script
  • Change the database from ‘master’ to ‘VIM_VCDB’ in the dropdown on the top bar
  • Press ‘Execute’
  • Evaluate the deleted rows, make sure it’s not more than you’d expect (ok, I didn’t do this)
  • Change
SET @DELETE_DATA = 0
to
SET @DELETE_DATA = 1
  • Press ‘Execute’ again.
  • Wait. Get a coffee. Get eight. It will eventually finish:
****************** SUMMARY *******************
Deleted 8400 rows from VPX_TASK table.
Deleted 2585209 rows from VPX_EVENT_ARG table.
Deleted 1662120 rows from VPX_EVENT table.
Deleted 0 rows from VPX_HIST_STAT1 table.
Deleted 0 rows from VPX_SAMPLE_TIME1 table.
Deleted 0 rows from VPX_HIST_STAT2 table.
Deleted 0 rows from VPX_SAMPLE_TIME2 table.
Deleted 0 rows from VPX_HIST_STAT3 table.
Deleted 0 rows from VPX_SAMPLE_TIME3 table.
Deleted 105331 rows from VPX_HIST_STAT4 table.
Deleted 373 rows from VPX_SAMPLE_TIME4 table.
  • Start VCenter Server. Wait. Try and connect. Hope. Pray.
  • Connect to VCenter Server
  • From the client, press Ctrl-Shift-I
  • Go to ‘Database Retention Policy’, and enable it.

Hopefully this will save someone a bit of googling.