Proxmox Backups – zstd and zpaq tested with astonishing results, 100% saving over multiple backups!

Standard

This post is in progress, and currently only shows the raw data I have collected this weekend on my Proxmox Backups, testing between zstd and zpaq. I will be doing a full write up over the next few days, so check back for that 🙂

Proxmox Static Backups - Space Saving
Original Backups
   605M Oct 10 11:38 vzdump-qemu-101-2021_10_10-11_37_35.vma.zst
   884M Oct 10 11:38 vzdump-qemu-102-2021_10_10-11_38_05.vma.zst
  3650M Oct 10 11:40 vzdump-qemu-104-2021_10_10-11_38_39.vma.zst
   393M Oct 10 11:41 vzdump-qemu-105-2021_10_10-11_40_32.vma.zst
  2493M Oct 10 15:35 vzdump-qemu-200-2021_10_10-15_34_05.vma.zst
 12500M Oct 10 15:42 vzdump-qemu-201-2021_10_10-15_35_59.vma.zst
  7164M Oct 10 15:46 vzdump-qemu-202-2021_10_10-15_42_01.vma.zst
   730M Oct 10 11:42 vzdump-qemu-203-2021_10_10-11_41_07.vma.zst
  2282M Oct 10 11:43 vzdump-qemu-204-2021_10_10-11_42_12.vma.zst
   681M Oct 10 11:44 vzdump-qemu-300-2021_10_10-11_43_48.vma.zst
  2215M Oct 10 11:46 vzdump-qemu-400-2021_10_10-11_44_51.vma.zst
   469M Oct 10 11:46 vzdump-qemu-999-2021_10_10-11_46_19.vma.zst
34,066M Total (All Files)

Backups archived and compressed with ZPAQ v7.15 (flags: -m5) 
Highest Compression Settings
  379M Oct 14 00:43 vzdump-qemu-101.zpaq
  639M Oct 14 04:31 vzdump-qemu-102.zpaq
 2997M Oct 13 17:11 vzdump-qemu-104.zpaq
  259M Oct 16 00:03 vzdump-qemu-105.zpaq
 1700M Oct 14 19:14 vzdump-qemu-200.zpaq
 9337M Oct 13 05:02 vzdump-qemu-201.zpaq
 5387M Oct 12 03:17 vzdump-qemu-202.zpaq
  496M Oct 13 23:41 vzdump-qemu-203.zpaq
 1546M Oct 15 02:22 vzdump-qemu-204.zpaq
  384M Oct 15 14:43 vzdump-qemu-300.zpaq
 1471M Oct 13 15:23 vzdump-qemu-400.zpaq
24,595M Total (All Files)

Compression Saving (without accounting for future deduplication savings)
9,471M compared to zstd compression.

---
Adding a second backup to each file, to test deduplication

Rough size of two total backups using zstd compression:
68,132M

   413M Oct 16 12:15 vzdump-qemu-101.zpaq
   680M Oct 16 12:18 vzdump-qemu-102.zpaq
  3113M Oct 16 12:24 vzdump-qemu-104.zpaq
   283M Oct 16 12:27 vzdump-qemu-105.zpaq
  1775M Oct 16 12:32 vzdump-qemu-200.zpaq
 12587M Oct 16 14:37 vzdump-qemu-201.zpaq
  7741M Oct 16 16:08 vzdump-qemu-202.zpaq
   537M Oct 16 16:14 vzdump-qemu-203.zpaq
  1619M Oct 16 16:21 vzdump-qemu-204.zpaq
   451M Oct 16 16:36 vzdump-qemu-300.zpaq
  1782M Oct 16 16:56 vzdump-qemu-400.zpaq
   313M Oct 16 17:16 vzdump-qemu-999.zpaq
   
Total size of 2 full backups of each VM with zpaq and deduplication:
30,981M
Saving over two zstd backups: 37,151M

CASCAIDr – A technically inept charity organisation that’s accepting donations but no new referrals.

Standard

So, as many people know we are struggling to get our appropriate rights respected by the care system in the UK. We thought we’d found at least a partial answer for getting legal information, advice and guidance. After looking online for feedback about CASCAIDr we thought we’d give them a try.

After contacting them after having difficulties filling in their referral form and going through everything with a woman on the phone, we get an email from the director which wasn’t ordered in any way that made sense to us. She said that they would be unable to help because they were in ‘special measures’ and I asked for clarification of what that meant and if what she’d written previously could be explained in a clearer way.

I did not receive a reply to my email, so I sent another email asking for the document they had referenced and whether it could be made clearer what she meant. The reply I received was “I cannot explain it again. We haven’t got the capacity to make it make sense, and it can’t be made any simpler.”

We have also been added to their marketing campaign emails without consent or opt-in.

CASCAIDr’s Problem

We have now figured out what the problem is – On their website, they have a pop-up that is hosted in the same way as an advertisement pop-up would be and is both blocked by ad blockers and assistive software to help disabled people read websites. It also doesn’t appear if you have JavaScript disabled.

I have found my entire experience with this organisation, past the first referral phone call to be very poor and I would perhaps get people think twice about making a donation to this organisation as they have not made it clear for people with ad blockers, assistive technology or those with JavaScript disabled to see that they actually won’t help anyone in a situation more complicated than a single letter (or approximately an hour’s work)

We have however now solved the problem. They do indeed have an alert on their page showing that they don’t accept new referrals, however it is not displayed in all circumstances. Here’s how the referral page looks if you’re using assistive software, an ad blocker or have JavaScript disabled:

CASCAIDr as it would appear to people with disability software, ad blockers or security conscious people that have JavaScript disabled for whatever reason.
CASCAIDr as it would appear to people with disability software, ad blockers or security conscious people that have JavaScript disabled for whatever reason.

Here is a screenshot what it looks like to someone without disability software, an ad blocker and those that still have JavaScript enabled.

CASCAIDr with disability software disabled and JavaScript enabled, without ad blocking software.
— CASCAIDr with disability software disabled and JavaScript enabled, without ad blocking software.

This is incredibly damning of CASCAIDr, they are a charity that primarily focuses on working with people online, yet this shows a complete degree of technical incompetence. They even have their ARIA tags set to ‘Hidden’ for this content, so anyone using a screen reader or other assistive software wouldn’t have a chance to see it and there’s something very suspicious about doing it this way.

If you’ve experienced any issues with CASCAIDr, do let us know, either as a comment on this article or you can contact me using the email address on the left or if you can’t use that, you can use the contact form here.

HTTP Status Endpoint for UptimeKuma for PHP websites like WordPress

Standard

So after I saw TechnoTim’s latest video showcasing a project called Uptime Kuma, he shows how to setup and use Uptime Kuma for monitoring websites and network services running under TCP. One of the problems with monitoring a website, such as one powered with WordPress, if you don’t use decent caching, the uptime check can put a bit of a load on your server, and it will have a longer uptime.

There’s two obvious answers. Setup and use caching or make a dedicated status endpoint for your website. This post looks at the second option, which will also work if you’re developing a website as you don’t want to use a cache while you’re still working on things, yet don’t want to cause unnecessary load on your server for a simple uptime check.

I wrote a simple PHP script for the task

<?php 
header('Content-Type: application/json');
$check_key = $_GET['key'];
if ($check_key == "SEKRET") { 
$http_status_code = http_response_code();
$current_time = date("c");
$response["success"] = true;
$response["http-status-code"] = $http_status_code;
$response["server-time"] = $current_time;
} else {
$response["success"] = false;
$response["authentication"] = "failed";
}
echo json_encode($response);
?>

You can add this to your server, and even set a secret key in case you don’t want other people to be able to access it. You would access it with status.php?key=SEKRET (in the case of my example). If like me, you’d like to make the URLs a bit prettier, you can use htaccess to setup a rewrite rule to hide the php file extension.

RewriteRule ^status$ /status.php?key=none [QSA,L]
RewriteRule ^status/([0-9a-zA-Z-]*) /status.php?key=$1 [QSA,L]

You need to add this somewhere in the rewrite section of your .htaccess file, somewhere near the top.

Then you can add the URL to Uptime Kuma, such as example.net/status/SEKRET (if you haven’t changed your key!) with a keyword, and search for “success”:true

Happy uptime! 🙂

Priory Care Group are still Disgusting and Neglectful in 2021!

Standard

The Priory Care Group who claim they “aspire to deliver the highest quality of care in the UK across our range of services, which include acute mental healthcare, addiction treatment and low and medium secure facilities.”

Priory, or No Priority?

The chief operations officer for Priory Care Group, a Rebekah Cresswell is responsible for “Priory Adult Care”, and has been quoted as saying (on Care Talk’s What Keeps Me Awake At Night?) “If it is not good enough for my family, then it is not good enough for one of our service users.” – However I have challenged her words with Priory Group and they just do not ring true. Would Rebekah be happy as a client of her own service?

Continue reading

PSXMinimise

Standard

So it was late one night and I couldn’t sleep, which is nothing abnormal for me and I got thinking. Is there any way I can improve the compression ratio of my collection of PS1 games, or was 7zip with LZMA the de-facto and best option? The truth it turns out is a little more complicated.

Updated 2021-07-15: Article finished, downloads and final conclusions added!

Continue reading

MSMTP Installation for System Mail

Standard

For getting email alerts and statuses from my servers, I use MSMTP to connect to an email address in the cloud, rather than dealing with setting up a full MTA.

The configuration is very easy in comparison to running something full blown like exim.

Continue reading

Elegoo Display and Stolen Code

Standard

So I bought myself another one of the Elegoo 2.8″ touchscreen LCD modules recently, and decided to have a deeper dive into the code. I looked through the examples and I thought “This looks very similar to code from Adafruit.”

You can download the original “tutorial” files from Elegoo directly at https://www.elegoo.com/pages/arduino-kits-support-files if you’d like to take a look for yourself.

[Update #1 added at 22:12 below]

If you’re in a rush and just want a better way to do this, use the > give me the code < link.

Continue reading

Getting PCI Passthrough working on G6 HP Proliant Servers with Proxmox

Standard

A better way to do this – Update: 2021-06-28

Someone has put together a patched kernel for Proxmox to let people run passthrough on their HP servers, it’s available on GitHub as a .deb download for direct install or you can build it from their sources yourself. Visit kiler129’s relax-intel-rmrr repository for a better way to do this 🙂

The rest of this post is included as an archive.

Continue reading

New Extensions for Email Blocking

Standard

I’ve since added a new amount of file extensions that I would recommend that people running mail-servers also block.

Originally I only blocked a few attachments.

ad[ep]|ba[st]|chm|cmd|com|cpl|crt|eml|exe|hlp|hta|in[fs]|isp|jse?|lnk|md[be]|ms[cipt]|pcd|pif|reg|scr|sct|shs|url|vb[se]|ws[fhc]

However now I’ve added a few more based on suggestions from various sources including extensions that Microsoft recommends to block for users of their Exchange server.

ace|ad[dp]|ani|app|appcontent-ms|appref-ms|as[px]|aspx|ba[st]|cdxml|cer|chm|cmd|cnt|com|cpl|crt|csh|der|diagcab|docm|eml|exe|fxp|gadget|grp|hlp|hpj|ht[acm]|html|in[fsp]|its|jar|jnlp|js|jse?|ksh|lnk|ma[dfgmqrstuvw]|mcf|md[abetwz]|mht|mhtml|ms[chptu]|msh1|msh1xml|msh2|msh2xml|mshxml|msi|ops|osd|pcd|pif|pl|plg|prf|prg|printerexport|ps1|ps1.xml|ps1xml|ps2|ps2xml|psc[12]|psd1|psdm1|psdm1cdxml|pssc|pst|py|py[cowz]|pyzw|reg|sc[frt]|settingcontent-ms|sh[bs]|theme|tmp|udl|url|vb|vb[eps]|vs[stw]|vsmacros|webpnp|website|ws|ws[cfh]|wsb|xbap|xll|xlsm|xml|xn|xnk

It should be relatively easy to copy the above into a regular expression suitable for your mail environment. If you think there’s a way I can optimise this list, please let me know 🙂

Note, this list used to contain the following, but I managed to optimise the expressions and remove duplicates, so the original is as follows (for reference)

ace|ad[dp]|ani|app|asp|aspx|asx|ba[st]|cer|chm|cmd|cnt|com|cpl|crt|csh|der|docm|exe|eml|fxp|gadget|hlp|hpj|ht[ac]|in[fsp]|its|jar|js|jse?|ksh|lnk|mad|maf|mag|mam|maq|mar|mas|mat|mau|mav|maw|md[abetwz]|mht|mhtml|ms[ch]|msh1|msh1xml|msh2|msh2xml|mshxml|msi|msp|mst|ops|osd|pcd|pif|plg|prf|prg|ps1|ps1xml|ps2|ps2xml|psc1|psc2|pst|reg|scf|sc[rt]|sh[bs]|tmp|url|vb|vb[eps]|vsmacros|vss|vst|vsw|ws|ws[cfh]|xml|xlsm|py|py[cowz]|pyzw|ps1|ps1.xml|ps2|ps2xml|psc[12]|psd1|psdm1|cdxml|pssc|appref-ms|udl|wsb|cer|crt|der|jar|jnlp|appcontent-ms|settingcontent-ms|cnt|hpj|website|webpnp|mcf|printerexport|pl|theme|xbap|xll|xnk|msu|diagcab|grp|pst|ps1xml|ps2|ps2xml|psc[12]|psd1|psdm1cdxml|pssc|appref-ms|udl|wsb|xbap|xll|xn

HiFiPi – A High Resolution Audio Player

Standard

This is a quick post to showcase our HiFiPi. I’ll try and update it in the future, but knowing me I’ll probably forget.

The software I used is called Volumio and it’s available here. They have some additional features available as a subscription service which you might like, but the core software is open source and free. They also sell pre-made high quality devices if you’re not into DIY 🙂

I seem to remember having a couple of issues getting it configured the way I wanted it, but I didn’t document any of it, so that will have to wait until I can figure it out. Overall nothing too taxing though.