PSXMinimise

Standard

This article is a work in progress (see notes at the end)

So it was late one night and I couldn’t sleep, which is nothing abnormal for me and I got thinking. Is there any way I can improve the compression ratio of my collection of PS1 games, or was 7zip with LZMA the de-facto and best option? The truth it turns out is a little more complicated.

Finding The ‘Best’ Compression

I’ve tried numerous compression formats in the search for the best one for PlayStation games. I needed a test game to find the best archive solution for the game data. I would tackle the audio content of the discs later. I grabbed my backup copy of Oddworld: Abe’s Oddysee (one of my favourite games) and tried all the regular formats that PeaZip could handle in their “Best” modes.

Compression FormatOptionsCompression TimeResulting Filesize
None685,461 KB
7ZipUltra2 minutes403,753 KB
ARC (FreeArc)94 minutes289,820 KB
Brotli935 minutes417,106 KB
BZip2Ultra16 minutes512,282 KB
GZipUltra7 minutes514,581 KB
XZUltra3 minutes403,755 KB
ZIPUltra6 minutes514,582 KB
Zstd193 minutes447,022 KB
ZPAQUltra34 minutes409,185 KB

Up until I saw the FreeArc compression sizes, it would seem that 7zip ultra compression was the best for PS1 data. At first, I thought there’d been some kind of error and the archive couldn’t have been as small as it became. However, after some checking I finally saw that the result was indeed accurate and it did give an excellent compression ratio of PlayStation 1 game data.

Sub-channel Data

A few years ago, I used to visit the EmuParadise website, and I recall that they used to compress their bin files with a tool called ECM. Armed with this vague piece of information I started searching the web for information about it. I managed to find the ECM Tools package (on archive.org) written in 2002 by Neill Corlett that is used to encode and decode files. These tools either add or remove sub-channel error correction data, or “Error Code Modelling” on a CD image. Removing this information will save some data from an image file and it can be added back in such a way that the file is an identical binary.

FilenameOriginal SizeSize (after ECM Encoding)
Destruction Derby 2 (Europe) (Track 01).bin71,603 KB64,141 KB
Dino Crisis (Europe) (Track 1).bin378,702 KB340,398 KB
Driver (Europe).bin729,961 KB651,308 KB
Grand Theft Auto (Europe) (Track 01).bin91,673 KB81,870 KB
MediEvil (Europe) (Track 1).bin516,356 KB478,157 KB
Oddworld – Abe’s Oddysee (Europe).bin685,461 KB612,203 KB
Tomb Raider II (Europe) (Track 01).bin224,530 KB218,150 KB

So it seems it is rather worth processing the ECM data, with all the games I’ve tested so far you get some kind of saving and with others you get a substantial saving.

Audio Compression

To compress the audio portions of the PS1 games, I decided to use FLAC, the free lossless audio encoder. I used this because the input files you pass to FLAC when decoded will be exactly the same as the original files. In a typical PS1 game, Track 01.bin is usually the game data, and if there are any other tracks (Track 02.bin, etc) they are CDDA files, which you can compress using FLAC if you tell it how these raw files can be processed.

flac -8 -V --force-raw-format --endian=little --channels=2 --bps=16 --sample-rate=44100 --sign=signed *.bin

An explanation of these options is that -8 sets the maximum level of FLAC compression and -V makes FLAC verify the files it has written after they have been created. The –force-raw-format argument instructs the command line version of FLAC to treat the input files as raw data. The –endian=little setting instructs FLAC of the byte order of the files. –channels=2 indicates that the audio stream is stereo. –bps=16 informs that the files are 16bit, –sample-rate=44100 is the standard audio sampling rate for a compact disc and finally –sign=signed tells FLAC that it’s a signed audio file. I am indebted to Cacovsky for the vital information on how raw CD audio can be converted to FLAC.

Putting It All Together

I had a few design goals for anything I created. It needed to rely on mostly open tools so that the techniques used are operating system agnostic and could fairly easily be ported to any other system and to use nothing that has encumbering license terms. Compression speed isn’t much of a factor as these files will really only be compressed once, however decompression speed shouldn’t be prohibitively slow. The methods used needed to be binary compatible so that the games were exactly the same after decompressing as they would be beforehand and I also wanted a system to verify that the files were indeed exactly the same as the originals.

From the results of compression earlier, FreeArc was definitely the obvious choice, given that it had a great improvement in compression ratios of the other archive formats and it didn’t take significantly longer to compress the data. Firstly I removed the ECM data from the first track and then processed the audio data from the other tracks so that it’s filesize is reduced using FLAC, the entire game can be packaged up in a .arc file.

Before doing this however, I needed a way to make sure that my compressed files would be the same as the originals when running the decompression process. On Linux there is a utility called sha256sum which will calculate a hash which can be compared against a file later to ensure that no changes have been made. On Windows there is a built in hash checker, however it doesn’t seem to be able to compare from a list of checksums in a single file and verify the data is intact. I am again indebted to Dave Benham who maintains HASHSUM.BAT at the DOSTips Forum. He created a script that is compatible with sha256sum but for Windows that works (mostly) in batch script.

The Scripts

So after a few sleepless nights, I’ve put together some utilities for automatically compressing and decompressing the games.

find . -maxdepth 1 -type d -exec bash -c "cd \"{}\" && sha256sum * > \"uncompressed.sha256\"" \;
@echo off
rem ------------------------------
rem PSXMinimise Compression Script
rem Version 0.17
rem Created by jcx
rem https://jcx.life/psxminimise
rem ------------------------------

IF EXIST "*(Track 01).bin" ( 
ecm *"(Track 01).bin"
del *"(Track 01).bin"
) ELSE (
IF EXIST "*(Track 1).bin" (
ecm *"(Track 1).bin"
del *"(Track 1).bin" 
) ELSE ( 
ecm *".bin"
GOTO compress
)
)

:audiocomp
flac -8 -V --force-raw-format --endian=little --channels=2 --bps=16 --sample-rate=44100 --sign=signed *.bin
del *.bin

:compress
for %%I in (.) do set CurrDirName=%%~nxI
freearc a -m9 -s "%CurrDirName%.arc" *.bin.ecm *.flac *.cue *.sha256

IF EXIST "*.bin.ecm" ( 
del *.bin.ecm
)
IF EXIST "*.flac" ( 
del *.flac
)
IF EXIST "*.cue" ( 
del *.cue
)
IF EXIST "*.sha256" ( 
del *.sha256
)
IF EXIST "*.bin" ( 
del *.bin
)

:end

My compression script is a bit more ugly than the unpacker, since I imagine people will unpack the games more often than they will pack them.

@echo off
rem ------------------------------
rem PSXMinimise Decompression Script
rem Version 0.37
rem Created by jcx
rem https://jcx.life/psxminimise
rem ------------------------------
color 5F
echo ---------------------------------------------------------
echo  ____  ______  ____  __ _       _           _
echo ^|  _ \/ ___\ \/ /  \/  (_)_ __ (_)_ __ ___ (_)___  ___
echo ^| ^|_) \___ \\  /^| ^|\/^| ^| ^| '_ \^| ^| '_ ` _ \^| / __^|/ _ \
echo ^|  __/ ___) /  \^| ^|  ^| ^| ^| ^| ^| ^| ^| ^| ^| ^| ^| ^| \__ \  __/
echo ^|_^|   ^|____/_/\_\_^|  ^|_^|_^|_^| ^|_^|_^|_^| ^|_^| ^|_^|_^|___/\___^|
echo.
echo ---------------------------------------------------------
echo       Created by jcx at https://jcx.life/psxminimise
echo                       Version 0.37
echo ---------------------------------------------------------
timeout /T 5
echo ---------------------------------------------------------
echo Unpacking .ARC
echo ---------------------------------------------------------
freearc e *.arc

echo ---------------------------------------------------------
echo Re-adding Error Code Modulation to Track 1
echo ---------------------------------------------------------
IF EXIST "*(Track 01).bin.ecm" ( 
unecm *"(Track 01).bin.ecm"
del *"(Track 01).bin.ecm"
goto audioextract
) ELSE (
IF EXIST "*(Track 1).bin.ecm" (
unecm *"(Track 1).bin.ecm"
del *"(Track 1).bin.ecm" 
goto audioextract
) ELSE ( 
unecm *".bin.ecm"
GOTO verify
)
)

:audioextract
echo ---------------------------------------------------------
echo Uncompressing CD audio tracks
echo ---------------------------------------------------------
flac -d --force-raw-format --endian little --sign=signed *.flac
rename *.raw *.bin
del *.flac
echo.
goto verify

:verify
echo ---------------------------------------------------------
echo Verifying Checksums
echo ---------------------------------------------------------
echo.
IF EXIST "*.bin.ecm" ( 
del *.bin.ecm
)
IF EXIST "*.flac" ( 
del *.flac
)
call hashsum.bat /c /nh /ns uncompressed.sha256 

	IF %ERRORLEVEL% EQU 0 GOTO success
GOTO error

:success
rem cls
IF EXIST "*.arc" ( 
del *.arc
)
color 2F
echo ---------------------------------------------------------
echo  ____  ______  ____  __ _       _           _
echo ^|  _ \/ ___\ \/ /  \/  (_)_ __ (_)_ __ ___ (_)___  ___
echo ^| ^|_) \___ \\  /^| ^|\/^| ^| ^| '_ \^| ^| '_ ` _ \^| / __^|/ _ \
echo ^|  __/ ___) /  \^| ^|  ^| ^| ^| ^| ^| ^| ^| ^| ^| ^| ^| ^| \__ \  __/
echo ^|_^|   ^|____/_/\_\_^|  ^|_^|_^|_^| ^|_^|_^|_^| ^|_^| ^|_^|_^|___/\___^|
echo.
echo ---------------------------------------------------------
echo       Created by jcx at https://jcx.life/psxminimise
echo                       Version 0.37
echo ---------------------------------------------------------
echo Unpacking has been completed successfully.
echo.
echo.
timeout /T 5
GOTO END

:error
color 4F
echo ---------------------------------------------------------
echo An error %ERRORLEVEL% occured.
echo ---------------------------------------------------------
pause
GOTO END
:end

Conclusion

GameOriginal Filesize7zip Filesize7z PercentagePSXMinimisePSXMinimise Percentrage
Destruction Derby 2620 MB533 MB85 %393 MB63 %
Dino Crisis405 MB202 MB49 %170 MB41. %
Driver712 MB357 MB50 %298 MB41 %
Grand Theft Auto714 MB612 MB85 %452 MB63 %
MediEvil535 MB322 MB60 %283 MB52 %
Oddworld: Abe’s Oddysee669 MB387 MB57 %241 MB36 %
Tomb Raider II708 MB460 MB64 %317 MB44 %
WipEout 2097681 MB566 MB83 %413 MB60 %
Average630.5 MB429 MB67.22 %320 MB50.60 %
Average Saving200 MB32.77 %309 MB49.40 %

You could get smaller game files if you removed some of the games content (a process known as “Ripping” using something like PocketISO to remove audio tracks and FMV sequences from games, however I wanted the games to be intact to their original release, and I think this has done quite a good job.

Note: This article isn’t completely finished, when writing I got rather tired and haven’t finished it off yet! Downloads and more to come soon! ~ jcx

MSMTP Installation for System Mail

Standard

For getting email alerts and statuses from my servers, I use MSMTP to connect to an email address in the cloud, rather than dealing with setting up a full MTA.

The configuration is very easy in comparison to running something full blown like exim.

To install on Debian 10, I enabled debian-backports and then installed the latest version from there.

apt-get install msmtp/buster-backports msmtp-mta/buster-backports

Configuration file for MSMTP:

# Set default values for all following accounts.
defaults
port 465
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
tls_starttls off
aliases /etc/aliases


account system
host email-host.example.com
from system@email-host.example.com
auth on
user system@email-host.example.com
password hackme

# Set a default account
account default : system

Required changes in /etc/aliases

postmaster: root
nobody: root
hostmaster: root
webmaster: root
www: root
root: system@email-host.example.com
default: system@email-host.example.com

Elegoo Display and Stolen Code

Standard

So I bought myself another one of the Elegoo 2.8″ touchscreen LCD modules recently, and decided to have a deeper dive into the code. I looked through the examples and I thought “This looks very similar to code from Adafruit.”

You can download the original “tutorial” files from Elegoo directly at https://www.elegoo.com/pages/arduino-kits-support-files if you’d like to take a look for yourself.

[Update #1 added at 22:12 below]

If you’re in a rush and just want a better way to do this, use the > give me the code < link.

The code was the same. The Elegoo_GFX library was an exact copy of the Adafruit GFX library, just with a global find and replace “Adafruit” for Elegoo. You can test this by changing the library definition in the code to use the Adafruit one and everything will still work.

One of the problems I had though, I could not figure out what they had changed to Adafruit’s TFTLCD library other than making it look as if they had written all the code themselves.

It took me the best part of an afternoon to finally figure out exactly how it worked. They have only made (from what I can tell) a single change to the library under definitions for the Arduino Mega 2560 in the code.

In the examples, it looks as if they use readID(); to get the chip Identifier that they use, however later on in the example they just set a static identifier. I know this because the readID(); function actually returns zero and they just statically set identifier=0x9341; to make everything work.

The only differences between Adafruit’s libraries and Elegoo’s are in the Elegoo_TFTLCD library file pin_magic.h at line 165.

If you want to make the LCD work with the 1.0.3 of the Adafruit TFTLCD Library, you can change lines 206 to 219 (which are) :

#define write8inline(d) 
  {
    PORTA = (d);
    WR_STROBE;
  }
#define read8inline(result)
  {
    RD_ACTIVE;
    DELAY7;
    result = PINA;
    RD_IDLE;
  }
#define setWriteDirInline() DDRA = 0xff
#define setReadDirInline() DDRA = 0
		#define write8inline(d) {\                        
		PORTH &= ~(0x78);\
  	PORTH |= ((d&0xC0) >> 3) | ((d&0x3) << 5);\
  	PORTE &= ~(0x38);\
  	PORTE |= ((d & 0xC) << 2) | ((d & 0x20) >> 2);\
  	PORTG &= ~(0x20);\
  	PORTG |= (d & 0x10) << 1; \ 
    WR_STROBE; }
  #define read8inline(result) { RD_ACTIVE; DELAY7; result = (PINH & 0x60) >> 5;result |= (PINH & 0x18) << 3;result |= (PINE & 0x8) << 2;result |= (PINE & 0x30) >> 2;result |= (PING & 0x20) >> 1;RD_IDLE;}
  #define setWriteDirInline() { DDRH |= 0x78;DDRE |= 0x38;DDRG |= 0x20; }
  #define setReadDirInline()  { DDRH &= ~0x78;DDRE &= ~0x38;DDRG &= ~(0x20); }

The only other changes you need to make to your code are when you start the tft.begin, you need to specify a fixed identifier, for example the following code (based on the graphicstest example)

// IMPORTANT: Adafruit_TFTLCD LIBRARY MUST BE SPECIFICALLY
// CONFIGURED FOR EITHER THE TFT SHIELD OR THE BREAKOUT BOARD.
// SEE RELEVANT COMMENTS IN Adafruit_TFTLCD.h FOR SETUP.

#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_TFTLCD.h> // Hardware-specific library
#define LCD_CS A3 // Chip Select goes to Analog 3
#define LCD_CD A2 // Command/Data goes to Analog 2
#define LCD_WR A1 // LCD Write goes to Analog 1
#define LCD_RD A0 // LCD Read goes to Analog 0

#define LCD_RESET A4 // Can alternately just connect to Arduino's reset pin

// Assign human-readable names to some common 16-bit color values:
#define	BLACK   0x0000
#define	BLUE    0x001F
#define	RED     0xF800
#define	GREEN   0x07E0
#define CYAN    0x07FF
#define MAGENTA 0xF81F
#define YELLOW  0xFFE0
#define WHITE   0xFFFF

Adafruit_TFTLCD tft(LCD_CS, LCD_CD, LCD_WR, LCD_RD, LCD_RESET);

void setup(void) {
  Serial.begin(9600);
  Serial.println(F("TFT LCD test"));

  Serial.print("TFT size is "); Serial.print(tft.width()); Serial.print("x"); Serial.println(tft.height());

  tft.reset();
  tft.begin(0x9341);

  Serial.println(F("Benchmark                Time (microseconds)"));

  Serial.print(F("Screen fill              "));
  Serial.println(testFillScreen());
  delay(500);

  Serial.print(F("Text                     "));
  Serial.println(testText());
  delay(3000);

  Serial.print(F("Lines                    "));
  Serial.println(testLines(CYAN));
  delay(500);

  Serial.print(F("Horiz/Vert Lines         "));
  Serial.println(testFastLines(RED, BLUE));
  delay(500);

  Serial.print(F("Rectangles (outline)     "));
  Serial.println(testRects(GREEN));
  delay(500);

  Serial.print(F("Rectangles (filled)      "));
  Serial.println(testFilledRects(YELLOW, MAGENTA));
  delay(500);

  Serial.print(F("Circles (filled)         "));
  Serial.println(testFilledCircles(10, MAGENTA));

  Serial.print(F("Circles (outline)        "));
  Serial.println(testCircles(10, WHITE));
  delay(500);

  Serial.print(F("Triangles (outline)      "));
  Serial.println(testTriangles());
  delay(500);

  Serial.print(F("Triangles (filled)       "));
  Serial.println(testFilledTriangles());
  delay(500);

  Serial.print(F("Rounded rects (outline)  "));
  Serial.println(testRoundRects());
  delay(500);

  Serial.print(F("Rounded rects (filled)   "));
  Serial.println(testFilledRoundRects());
  delay(500);

  Serial.println(F("Done!"));
}

void loop(void) {
  for(uint8_t rotation=0; rotation<4; rotation++) {
    tft.setRotation(rotation);
    testText();
    delay(2000);
  }
}

unsigned long testFillScreen() {
  unsigned long start = micros();
  tft.fillScreen(BLACK);
  tft.fillScreen(RED);
  tft.fillScreen(GREEN);
  tft.fillScreen(BLUE);
  tft.fillScreen(BLACK);
  return micros() - start;
}

unsigned long testText() {
  tft.fillScreen(BLACK);
  unsigned long start = micros();
  tft.setCursor(0, 0);
  tft.setTextColor(WHITE);  tft.setTextSize(1);
  tft.println("Hello World!");
  tft.setTextColor(YELLOW); tft.setTextSize(2);
  tft.println(1234.56);
  tft.setTextColor(RED);    tft.setTextSize(3);
  tft.println(0xDEADBEEF, HEX);
  tft.println();
  tft.setTextColor(GREEN);
  tft.setTextSize(5);
  tft.println("Groop");
  tft.setTextSize(2);
  tft.println("I implore thee,");
  tft.setTextSize(1);
  tft.println("my foonting turlingdromes.");
  tft.println("And hooptiously drangle me");
  tft.println("with crinkly bindlewurdles,");
  tft.println("Or I will rend thee");
  tft.println("in the gobberwarts");
  tft.println("with my blurglecruncheon,");
  tft.println("see if I don't!");
  return micros() - start;
}

unsigned long testLines(uint16_t color) {
  unsigned long start, t;
  int           x1, y1, x2, y2,
                w = tft.width(),
                h = tft.height();

  tft.fillScreen(BLACK);

  x1 = y1 = 0;
  y2    = h - 1;
  start = micros();
  for(x2=0; x2<w; x2+=6) tft.drawLine(x1, y1, x2, y2, color);
  x2    = w - 1;
  for(y2=0; y2<h; y2+=6) tft.drawLine(x1, y1, x2, y2, color);
  t     = micros() - start; // fillScreen doesn't count against timing

  tft.fillScreen(BLACK);

  x1    = w - 1;
  y1    = 0;
  y2    = h - 1;
  start = micros();
  for(x2=0; x2<w; x2+=6) tft.drawLine(x1, y1, x2, y2, color);
  x2    = 0;
  for(y2=0; y2<h; y2+=6) tft.drawLine(x1, y1, x2, y2, color);
  t    += micros() - start;

  tft.fillScreen(BLACK);

  x1    = 0;
  y1    = h - 1;
  y2    = 0;
  start = micros();
  for(x2=0; x2<w; x2+=6) tft.drawLine(x1, y1, x2, y2, color);
  x2    = w - 1;
  for(y2=0; y2<h; y2+=6) tft.drawLine(x1, y1, x2, y2, color);
  t    += micros() - start;

  tft.fillScreen(BLACK);

  x1    = w - 1;
  y1    = h - 1;
  y2    = 0;
  start = micros();
  for(x2=0; x2<w; x2+=6) tft.drawLine(x1, y1, x2, y2, color);
  x2    = 0;
  for(y2=0; y2<h; y2+=6) tft.drawLine(x1, y1, x2, y2, color);

  return micros() - start;
}

unsigned long testFastLines(uint16_t color1, uint16_t color2) {
  unsigned long start;
  int           x, y, w = tft.width(), h = tft.height();

  tft.fillScreen(BLACK);
  start = micros();
  for(y=0; y<h; y+=5) tft.drawFastHLine(0, y, w, color1);
  for(x=0; x<w; x+=5) tft.drawFastVLine(x, 0, h, color2);

  return micros() - start;
}

unsigned long testRects(uint16_t color) {
  unsigned long start;
  int           n, i, i2,
                cx = tft.width()  / 2,
                cy = tft.height() / 2;

  tft.fillScreen(BLACK);
  n     = min(tft.width(), tft.height());
  start = micros();
  for(i=2; i<n; i+=6) {
    i2 = i / 2;
    tft.drawRect(cx-i2, cy-i2, i, i, color);
  }

  return micros() - start;
}

unsigned long testFilledRects(uint16_t color1, uint16_t color2) {
  unsigned long start, t = 0;
  int           n, i, i2,
                cx = tft.width()  / 2 - 1,
                cy = tft.height() / 2 - 1;

  tft.fillScreen(BLACK);
  n = min(tft.width(), tft.height());
  for(i=n; i>0; i-=6) {
    i2    = i / 2;
    start = micros();
    tft.fillRect(cx-i2, cy-i2, i, i, color1);
    t    += micros() - start;
    // Outlines are not included in timing results
    tft.drawRect(cx-i2, cy-i2, i, i, color2);
  }

  return t;
}

unsigned long testFilledCircles(uint8_t radius, uint16_t color) {
  unsigned long start;
  int x, y, w = tft.width(), h = tft.height(), r2 = radius * 2;

  tft.fillScreen(BLACK);
  start = micros();
  for(x=radius; x<w; x+=r2) {
    for(y=radius; y<h; y+=r2) {
      tft.fillCircle(x, y, radius, color);
    }
  }

  return micros() - start;
}

unsigned long testCircles(uint8_t radius, uint16_t color) {
  unsigned long start;
  int           x, y, r2 = radius * 2,
                w = tft.width()  + radius,
                h = tft.height() + radius;

  // Screen is not cleared for this one -- this is
  // intentional and does not affect the reported time.
  start = micros();
  for(x=0; x<w; x+=r2) {
    for(y=0; y<h; y+=r2) {
      tft.drawCircle(x, y, radius, color);
    }
  }

  return micros() - start;
}

unsigned long testTriangles() {
  unsigned long start;
  int           n, i, cx = tft.width()  / 2 - 1,
                      cy = tft.height() / 2 - 1;

  tft.fillScreen(BLACK);
  n     = min(cx, cy);
  start = micros();
  for(i=0; i<n; i+=5) {
    tft.drawTriangle(
      cx    , cy - i, // peak
      cx - i, cy + i, // bottom left
      cx + i, cy + i, // bottom right
      tft.color565(0, 0, i));
  }

  return micros() - start;
}

unsigned long testFilledTriangles() {
  unsigned long start, t = 0;
  int           i, cx = tft.width()  / 2 - 1,
                   cy = tft.height() / 2 - 1;

  tft.fillScreen(BLACK);
  start = micros();
  for(i=min(cx,cy); i>10; i-=5) {
    start = micros();
    tft.fillTriangle(cx, cy - i, cx - i, cy + i, cx + i, cy + i,
      tft.color565(0, i, i));
    t += micros() - start;
    tft.drawTriangle(cx, cy - i, cx - i, cy + i, cx + i, cy + i,
      tft.color565(i, i, 0));
  }

  return t;
}

unsigned long testRoundRects() {
  unsigned long start;
  int           w, i, i2,
                cx = tft.width()  / 2 - 1,
                cy = tft.height() / 2 - 1;

  tft.fillScreen(BLACK);
  w     = min(tft.width(), tft.height());
  start = micros();
  for(i=0; i<w; i+=6) {
    i2 = i / 2;
    tft.drawRoundRect(cx-i2, cy-i2, i, i, i/8, tft.color565(i, 0, 0));
  }

  return micros() - start;
}

unsigned long testFilledRoundRects() {
  unsigned long start;
  int           i, i2,
                cx = tft.width()  / 2 - 1,
                cy = tft.height() / 2 - 1;

  tft.fillScreen(BLACK);
  start = micros();
  for(i=min(tft.width(), tft.height()); i>20; i-=6) {
    i2 = i / 2;
    tft.fillRoundRect(cx-i2, cy-i2, i, i, i/8, tft.color565(0, i, 0));
  }

  return micros() - start;
}

I’m waiting for Amazon to publish my product review, as I think this a totally unfair use of Adafruit’s source code and quite a clear ethical violation of the ethos of open source code. They could have easily forked the library and added their changes… Maybe it would have even been accepted up stream if people would find using similar displays with the Adafruit library on the Mega 2650.

Here’s a copy of my pending Amazon review.

This article is research in progress and may change as I make future discoveries. I plan to test the code on a regular Arduino Uno to see if it works on that or there’s any other trickery afoot.

[Update #1 – 22:12]: After much more research, I discovered the library MCUFRIEND_kbv. It’s included showBMP_not_Uno didn’t seem to work with the latest version of SdFat or on a Mega 2650, so I have updated this example (included below) to work mostly out of the box.

You don’t have to do any library edits like above, other than defining #define SPI_DRIVER_SELECT 2 within SdFatConfig.h (in your libraries folder), then you can put your 24bit BMP images on your SD card and load them when using this module without any of the massive amount of effort I went through today 🙂

// Modified from MCUFRIEND-kbv version 2.99 by jcx 2021-04-23
// Upload and play sketch for Arduino Mega 2560 and Elegoo 2.8" TFT LCD
// MCUFRIEND UNO shields have microSD on pins 10, 11, 12, 13
// The official <SD.h> library only works on the hardware SPI pins
// e.g. 11, 12, 13 on a Uno
// e.g. 50, 51, 52 on a Mega2560
// e.g. 74, 75, 76 on a Due
//
// if you are not using a UNO,  you must use Software SPI:
// install the latest version of the <SdFat.h> library with the Arduino Library Manager.
// edit the src/SdFatConfig.h file to #define SPI_DRIVER_SELECT 2
//
// copy all your BMP files to the root directory on the microSD with your PC
// (or another directory)

#include <SPI.h>             // f.k. for Arduino-1.5.2
#include <SdFat.h>           // Use the SdFat library
SdFat SD;
// Pin numbers in templates must be constants.
const uint8_t SOFT_MISO_PIN = 12;
const uint8_t SOFT_MOSI_PIN = 11;
const uint8_t SOFT_SCK_PIN  = 13;
// Chip select may be constant or RAM variable.
const uint8_t SD_CS_PIN = 10;

// SdFat software SPI template
SoftSpiDriver<SOFT_MISO_PIN, SOFT_MOSI_PIN, SOFT_SCK_PIN> softSpi;
// Speed argument is ignored for software SPI.
#if ENABLE_DEDICATED_SPI
#define SD_CONFIG SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(0), &softSpi)
#else  // ENABLE_DEDICATED_SPI
#define SD_CONFIG SdSpiConfig(SD_CS_PIN, SHARED_SPI, SD_SCK_MHZ(0), &softSpi)
#endif  // ENABLE_DEDICATED_SPI

SdFat32 sd;
File32 file;

#include <Adafruit_GFX.h>    // Hardware-specific library
#include <MCUFRIEND_kbv.h>

MCUFRIEND_kbv tft;

#define NAMEMATCH ""         // "" matches any name
//#define NAMEMATCH "tiger"    // *tiger*.bmp
#define PALETTEDEPTH   8     // support 256-colour Palette

char namebuf[32] = "/";   //BMP files in root directory
//char namebuf[32] = "/bitmaps/";  //BMP directory e.g. files in /bitmaps/*.bmp

File root;
int pathlen;

void setup()
{
    uint16_t ID;
    Serial.begin(9600);
    Serial.print("Show BMP files on TFT with ID:0x");
    ID = tft.readID();
    Serial.println(ID, HEX);
    if (ID == 0x0D3D3) ID = 0x9481;
    tft.begin(ID);
    tft.fillScreen(0x001F);
    tft.setTextColor(0xFFFF, 0x0000);
    bool good = SD.begin(SD_CONFIG);
    if (!good) {
        Serial.print(F("cannot start SD"));
        while (1);
    }
    root = SD.open(namebuf);
    pathlen = strlen(namebuf);
}

void loop()
{
    char *nm = namebuf + pathlen;
    File f = root.openNextFile();
    uint8_t ret;
    uint32_t start;
    if (f != NULL) {
        f.getName(nm, 32 - pathlen);
        f.close();
        strlwr(nm);
        if (strstr(nm, ".bmp") != NULL && strstr(nm, NAMEMATCH) != NULL) {
            Serial.print(namebuf);
            Serial.print(F(" - "));
            tft.fillScreen(0);
            start = millis();
            ret = showBMP(namebuf, 5, 5);
            switch (ret) {
                case 0:
                    Serial.print(millis() - start);
                    Serial.println(F("ms"));
                    delay(5000);
                    break;
                case 1:
                    Serial.println(F("bad position"));
                    break;
                case 2:
                    Serial.println(F("bad BMP ID"));
                    break;
                case 3:
                    Serial.println(F("wrong number of planes"));
                    break;
                case 4:
                    Serial.println(F("unsupported BMP format"));
                    break;
                case 5:
                    Serial.println(F("unsupported palette"));
                    break;
                default:
                    Serial.println(F("unknown"));
                    break;
            }
        }
    }
    else root.rewindDirectory();
}

#define BMPIMAGEOFFSET 54
#define BUFFPIXEL      20

uint16_t read16(File& f) {
    uint16_t result;         // read little-endian
    f.read(&result, sizeof(result));
    return result;
}

uint32_t read32(File& f) {
    uint32_t result;
    f.read(&result, sizeof(result));
    return result;
}

uint8_t showBMP(char *nm, int x, int y)
{
    File bmpFile;
    int bmpWidth, bmpHeight;    // W+H in pixels
    uint8_t bmpDepth;           // Bit depth (currently must be 24, 16, 8, 4, 1)
    uint32_t bmpImageoffset;    // Start of image data in file
    uint32_t rowSize;           // Not always = bmpWidth; may have padding
    uint8_t sdbuffer[3 * BUFFPIXEL];    // pixel in buffer (R+G+B per pixel)
    uint16_t lcdbuffer[(1 << PALETTEDEPTH) + BUFFPIXEL], *palette = NULL;
    uint8_t bitmask, bitshift;
    boolean flip = true;        // BMP is stored bottom-to-top
    int w, h, row, col, lcdbufsiz = (1 << PALETTEDEPTH) + BUFFPIXEL, buffidx;
    uint32_t pos;               // seek position
    boolean is565 = false;      //

    uint16_t bmpID;
    uint16_t n;                 // blocks read
    uint8_t ret;

    if ((x >= tft.width()) || (y >= tft.height()))
        return 1;               // off screen

    bmpFile = SD.open(nm);      // Parse BMP header
    bmpID = read16(bmpFile);    // BMP signature
    (void) read32(bmpFile);     // Read & ignore file size
    (void) read32(bmpFile);     // Read & ignore creator bytes
    bmpImageoffset = read32(bmpFile);       // Start of image data
    (void) read32(bmpFile);     // Read & ignore DIB header size
    bmpWidth = read32(bmpFile);
    bmpHeight = read32(bmpFile);
    n = read16(bmpFile);        // # planes -- must be '1'
    bmpDepth = read16(bmpFile); // bits per pixel
    pos = read32(bmpFile);      // format
    if (bmpID != 0x4D42) ret = 2; // bad ID
    else if (n != 1) ret = 3;   // too many planes
    else if (pos != 0 && pos != 3) ret = 4; // format: 0 = uncompressed, 3 = 565
    else if (bmpDepth < 16 && bmpDepth > PALETTEDEPTH) ret = 5; // palette 
    else {
        bool first = true;
        is565 = (pos == 3);               // ?already in 16-bit format
        // BMP rows are padded (if needed) to 4-byte boundary
        rowSize = (bmpWidth * bmpDepth / 8 + 3) & ~3;
        if (bmpHeight < 0) {              // If negative, image is in top-down order.
            bmpHeight = -bmpHeight;
            flip = false;
        }

        w = bmpWidth;
        h = bmpHeight;
        if ((x + w) >= tft.width())       // Crop area to be loaded
            w = tft.width() - x;
        if ((y + h) >= tft.height())      //
            h = tft.height() - y;

        if (bmpDepth <= PALETTEDEPTH) {   // these modes have separate palette
            //bmpFile.seek(BMPIMAGEOFFSET); //palette is always @ 54
            bmpFile.seek(bmpImageoffset - (4<<bmpDepth)); //54 for regular, diff for colorsimportant
            bitmask = 0xFF;
            if (bmpDepth < 8)
                bitmask >>= bmpDepth;
            bitshift = 8 - bmpDepth;
            n = 1 << bmpDepth;
            lcdbufsiz -= n;
            palette = lcdbuffer + lcdbufsiz;
            for (col = 0; col < n; col++) {
                pos = read32(bmpFile);    //map palette to 5-6-5
                palette[col] = ((pos & 0x0000F8) >> 3) | ((pos & 0x00FC00) >> 5) | ((pos & 0xF80000) >> 8);
            }
        }

        // Set TFT address window to clipped image bounds
        tft.setAddrWindow(x, y, x + w - 1, y + h - 1);
        for (row = 0; row < h; row++) { // For each scanline...
            // Seek to start of scan line.  It might seem labor-
            // intensive to be doing this on every line, but this
            // method covers a lot of gritty details like cropping
            // and scanline padding.  Also, the seek only takes
            // place if the file position actually needs to change
            // (avoids a lot of cluster math in SD library).
            uint8_t r, g, b, *sdptr;
            int lcdidx, lcdleft;
            if (flip)   // Bitmap is stored bottom-to-top order (normal BMP)
                pos = bmpImageoffset + (bmpHeight - 1 - row) * rowSize;
            else        // Bitmap is stored top-to-bottom
                pos = bmpImageoffset + row * rowSize;
            if (bmpFile.position() != pos) { // Need seek?
                bmpFile.seek(pos);
                buffidx = sizeof(sdbuffer); // Force buffer reload
            }

            for (col = 0; col < w; ) {  //pixels in row
                lcdleft = w - col;
                if (lcdleft > lcdbufsiz) lcdleft = lcdbufsiz;
                for (lcdidx = 0; lcdidx < lcdleft; lcdidx++) { // buffer at a time
                    uint16_t color;
                    // Time to read more pixel data?
                    if (buffidx >= sizeof(sdbuffer)) { // Indeed
                        bmpFile.read(sdbuffer, sizeof(sdbuffer));
                        buffidx = 0; // Set index to beginning
                        r = 0;
                    }
                    switch (bmpDepth) {          // Convert pixel from BMP to TFT format
                        case 24:
                            b = sdbuffer[buffidx++];
                            g = sdbuffer[buffidx++];
                            r = sdbuffer[buffidx++];
                            color = tft.color565(r, g, b);
                            break;
                        case 16:
                            b = sdbuffer[buffidx++];
                            r = sdbuffer[buffidx++];
                            if (is565)
                                color = (r << 8) | (b);
                            else
                                color = (r << 9) | ((b & 0xE0) << 1) | (b & 0x1F);
                            break;
                        case 1:
                        case 4:
                        case 8:
                            if (r == 0)
                                b = sdbuffer[buffidx++], r = 8;
                            color = palette[(b >> bitshift) & bitmask];
                            r -= bmpDepth;
                            b <<= bmpDepth;
                            break;
                    }
                    lcdbuffer[lcdidx] = color;

                }
                tft.pushColors(lcdbuffer, lcdidx, first);
                first = false;
                col += lcdidx;
            }           // end cols
        }               // end rows
        tft.setAddrWindow(0, 0, tft.width() - 1, tft.height() - 1); //restore full screen
        ret = 0;        // good render
    }
    bmpFile.close();
    return (ret);
}

[Give Me The Code:] If you’re too lazy efficient and just want some code for an Arduino Mega 2560 that can display an image from an SD card (a 24 bit BMP called image.bmp), draw some buttons on the screen and toggle something as an example, you’re in luck.

You need to install some libraries for this to work, MCUFRIEND_kbv, Adafruit_GFX, SdFat and Adafruit_Touchscreen. You might need to run the touch screen calibration sketch included with MCUFRIEND_kbv (TouchScreen_Calibr_native.ino) which will give you your calibration to just copy and paste into your code. Enough talk, give me the code! Righto!

// This Example file is based on examples from MCUFRIEND_kbv, Adafruit_GFX, SdFat and Adafruit_Touchscreen and was updated by jcx on 2021-04-23 with <3

#include <Adafruit_GFX.h>
#include <MCUFRIEND_kbv.h>
MCUFRIEND_kbv tft;
#include <TouchScreen.h>
#include "SdFat.h"
#define MINPRESSURE 200
#define MAXPRESSURE 1000

SdFat SD;
// Pin numbers in templates must be constants.
const uint8_t SOFT_MISO_PIN = 12;
const uint8_t SOFT_MOSI_PIN = 11;
const uint8_t SOFT_SCK_PIN  = 13;
// Chip select may be constant or RAM variable.
const uint8_t SD_CS_PIN = 10;

// SdFat software SPI template
SoftSpiDriver<SOFT_MISO_PIN, SOFT_MOSI_PIN, SOFT_SCK_PIN> softSpi;
// Speed argument is ignored for software SPI.
#if ENABLE_DEDICATED_SPI
#define SD_CONFIG SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(0), &softSpi)
#else  // ENABLE_DEDICATED_SPI
#define SD_CONFIG SdSpiConfig(SD_CS_PIN, SHARED_SPI, SD_SCK_MHZ(0), &softSpi)
#endif  // ENABLE_DEDICATED_SPI

SdFat32 sd;
File32 file;

// ALL Touch panels and wiring is DIFFERENT
// copy-paste results from TouchScreen_Calibr_native.ino
const int XP=8,XM=A2,YP=A3,YM=9; //240x320 ID=0x9341
const int TS_LEFT=913,TS_RT=118,TS_TOP=80,TS_BOT=896;

TouchScreen ts = TouchScreen(XP, YP, XM, YM, 300);

Adafruit_GFX_Button on_btn, off_btn, meh_btn;

int pixel_x, pixel_y;     //Touch_getXY() updates global vars
bool Touch_getXY(void)
{
    TSPoint p = ts.getPoint();
    pinMode(YP, OUTPUT);      //restore shared pins
    pinMode(XM, OUTPUT);
    digitalWrite(YP, HIGH);   //because TFT control pins
    digitalWrite(XM, HIGH);
    bool pressed = (p.z > MINPRESSURE && p.z < MAXPRESSURE);
    if (pressed) {
        pixel_x = map(p.x, TS_LEFT, TS_RT, 0, tft.width()); //.kbv makes sense to me
        pixel_y = map(p.y, TS_TOP, TS_BOT, 0, tft.height());
    }
    return pressed;
}

#define BLACK   0x0000
#define BLUE    0x001F
#define RED     0xF800
#define GREEN   0x07E0
#define CYAN    0x07FF
#define MAGENTA 0xF81F
#define YELLOW  0xFFE0
#define WHITE   0xFFFF

void setup(void)
{
    Serial.begin(9600);
      Serial.print(F("Initializing SD card..."));
  if (!SD.begin(SD_CONFIG)) {
    Serial.println(F("failed!"));
    return;
  }
  Serial.println(F("OK!"));

    uint16_t ID = tft.readID();
    Serial.print("TFT ID = 0x");
    Serial.println(ID, HEX);
    Serial.println("Calibrate for your Touch Panel");
    if (ID == 0xD3D3) ID = 0x9486; // write-only shield
    tft.begin(ID);
    tft.setRotation(0);            //PORTRAIT
    tft.fillScreen(BLACK);
    
  bmpDraw("image.bmp", 0,0);
  delay(4000);
    on_btn.initButton(&tft,  60, 200, 100, 40, WHITE, MAGENTA, BLACK, "ON", 2);
    off_btn.initButton(&tft, 180, 200, 100, 40, WHITE, MAGENTA, BLACK, "OFF", 2);
    //btn name, tft declaration, xpos, ypos, width, height, border colour, background color, text color
    meh_btn.initButton(&tft, 215, 25, 50, 50, WHITE, MAGENTA, BLACK, "MEH", 2);
    on_btn.drawButton(false);
    off_btn.drawButton(false);
    meh_btn.drawButton(false); // "PRESSED?"
    tft.fillRect(40, 80, 160, 80, RED);
}

/*  
 * updating multiple buttons from a list
 * 
 * anything more than two buttons gets repetitive
 * 
 * you can place button addresses in separate lists
 * e.g. for separate menu screens
 */

// Array of button addresses to behave like a list
Adafruit_GFX_Button *buttons[] = {&on_btn, &off_btn, &meh_btn, NULL};

/* update the state of a button and redraw as reqd
 *
 * main program can use isPressed(), justPressed() etc
 */
bool update_button(Adafruit_GFX_Button *b, bool down)
{
    b->press(down && b->contains(pixel_x, pixel_y));
    if (b->justReleased())
        b->drawButton(false);
    if (b->justPressed())
        b->drawButton(true);
    return down;
}

/* most screens have different sets of buttons
 * life is easier if you process whole list in one go
 */
bool update_button_list(Adafruit_GFX_Button **pb)
{
    bool down = Touch_getXY();
    for (int i = 0 ; pb[i] != NULL; i++) {
        update_button(pb[i], down);
    }
    return down;
}

/* compare the simplicity of update_button_list()
 */
void loop(void)
{
    update_button_list(buttons);  //use helper function
    if (on_btn.justPressed()) {
        tft.fillRect(40, 80, 160, 80, GREEN);
    }
    if (off_btn.justPressed()) {
        tft.fillRect(40, 80, 160, 80, RED);
    }
}
#define BUFFPIXEL 20

void bmpDraw(char *filename, int x, int y) {

  File     bmpFile;
  int      bmpWidth, bmpHeight;   // W+H in pixels
  uint8_t  bmpDepth;              // Bit depth (currently must be 24)
  uint32_t bmpImageoffset;        // Start of image data in file
  uint32_t rowSize;               // Not always = bmpWidth; may have padding
  uint8_t  sdbuffer[3*BUFFPIXEL]; // pixel in buffer (R+G+B per pixel)
  uint16_t lcdbuffer[BUFFPIXEL];  // pixel out buffer (16-bit per pixel)
  uint8_t  buffidx = sizeof(sdbuffer); // Current position in sdbuffer
  boolean  goodBmp = false;       // Set to true on valid header parse
  boolean  flip    = true;        // BMP is stored bottom-to-top
  int      w, h, row, col;
  uint8_t  r, g, b;
  uint32_t pos = 0, startTime = millis();
  uint8_t  lcdidx = 0;
  boolean  first = true;

  if((x >= tft.width()) || (y >= tft.height())) return;

  Serial.println();
  Serial.print(F("Loading image '"));
  Serial.print(filename);
  Serial.println('\'');
  // Open requested file on SD card
  bmpFile = SD.open(filename);
  if (bmpFile==NULL) {
    Serial.println(F("File not found"));
    return;
  }

  // Parse BMP header
  if(read16(bmpFile) == 0x4D42) { // BMP signature
    Serial.println(F("File size: ")); Serial.println(read32(bmpFile));
    (void)read32(bmpFile); // Read & ignore creator bytes
    bmpImageoffset = read32(bmpFile); // Start of image data
    Serial.print(F("Image Offset: ")); Serial.println(bmpImageoffset, DEC);
    // Read DIB header
    Serial.print(F("Header size: ")); Serial.println(read32(bmpFile));
    bmpWidth  = read32(bmpFile);
    bmpHeight = read32(bmpFile);
    if(read16(bmpFile) == 1) { // # planes -- must be '1'
      bmpDepth = read16(bmpFile); // bits per pixel
      Serial.print(F("Bit Depth: ")); Serial.println(bmpDepth);
      if((bmpDepth == 24) && (read32(bmpFile) == 0)) { // 0 = uncompressed

        goodBmp = true; // Supported BMP format -- proceed!
        Serial.print(F("Image size: "));
        Serial.print(bmpWidth);
        Serial.print('x');
        Serial.println(bmpHeight);

        // BMP rows are padded (if needed) to 4-byte boundary
        rowSize = (bmpWidth * 3 + 3) & ~3;

        // If bmpHeight is negative, image is in top-down order.
        // This is not canon but has been observed in the wild.
        if(bmpHeight < 0) {
          bmpHeight = -bmpHeight;
          flip      = false;
        }

        // Crop area to be loaded
        w = bmpWidth;
        h = bmpHeight;
        if((x+w-1) >= tft.width())  w = tft.width()  - x;
        if((y+h-1) >= tft.height()) h = tft.height() - y;

        // Set TFT address window to clipped image bounds
        tft.setAddrWindow(x, y, x+w-1, y+h-1);

        for (row=0; row<h; row++) { // For each scanline...
          // Seek to start of scan line.  It might seem labor-
          // intensive to be doing this on every line, but this
          // method covers a lot of gritty details like cropping
          // and scanline padding.  Also, the seek only takes
          // place if the file position actually needs to change
          // (avoids a lot of cluster math in SD library).
          if(flip) // Bitmap is stored bottom-to-top order (normal BMP)
            pos = bmpImageoffset + (bmpHeight - 1 - row) * rowSize;
          else     // Bitmap is stored top-to-bottom
            pos = bmpImageoffset + row * rowSize;
          if(bmpFile.position() != pos) { // Need seek?
            bmpFile.seek(pos);
            buffidx = sizeof(sdbuffer); // Force buffer reload
          }

          for (col=0; col<w; col++) { // For each column...
            // Time to read more pixel data?
            if (buffidx >= sizeof(sdbuffer)) { // Indeed
              // Push LCD buffer to the display first
              if(lcdidx > 0) {
                tft.pushColors(lcdbuffer, lcdidx, first);
                lcdidx = 0;
                first  = false;
              }
              bmpFile.read(sdbuffer, sizeof(sdbuffer));
              buffidx = 0; // Set index to beginning
            }

            // Convert pixel from BMP to TFT format
            b = sdbuffer[buffidx++];
            g = sdbuffer[buffidx++];
            r = sdbuffer[buffidx++];
            lcdbuffer[lcdidx++] = tft.color565(r,g,b);
          } // end pixel
        } // end scanline
        // Write any remaining data to LCD
        if(lcdidx > 0) {
          tft.pushColors(lcdbuffer, lcdidx, first);
        } 
        Serial.print(F("Loaded in "));
        Serial.print(millis() - startTime);
        Serial.println(" ms");
      } // end goodBmp
    }
  }

  bmpFile.close();
  if(!goodBmp) Serial.println(F("BMP format not recognized."));
}
// These read 16- and 32-bit types from the SD card file.
// BMP data is stored little-endian, Arduino is little-endian too.
// May need to reverse subscript order if porting elsewhere.

uint16_t read16(File& f) {
  uint16_t result;
  ((uint8_t *)&result)[0] = f.read(); // LSB
  ((uint8_t *)&result)[1] = f.read(); // MSB
  return result;
}

uint32_t read32(File& f) {
  uint32_t result;
  ((uint8_t *)&result)[0] = f.read(); // LSB
  ((uint8_t *)&result)[1] = f.read();
  ((uint8_t *)&result)[2] = f.read();
  ((uint8_t *)&result)[3] = f.read(); // MSB
  return result;
}

Hope this was all useful, and saves someone else their afternoon and evening 🙂 <3

Getting PCI Passthrough working on G6 HP Proliant Servers with Proxmox

Standard

This is a quick stub post to show how I finally got PCI pass-through working on my HP DL360 G6 server, since it would show an error such as

DMAR-IR: This system BIOS has enabled interrupt remapping interrupt remapping is being disabled.

If you actually tried to launch a machine with a PCI device attached, it would say:

Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

From what I can tell this is because the system uses RMRR to control fan speeds, and having these bits set causes the driver to disable IOMMU. There is a simple change you can make to the Proxmox kernel source code to get passed this error and use PCI devices with VMs without issue.

sed -i.new 's/device_is_rmrr_locked(dev)/false/' ./$UBUNTU_NAME/drivers/iommu/intel-iommu.c

After making that change and recompiling the kernel, it works perfectly fine, but the system will still say DMAR-IR interrupts have been disabled, however it will work fine.

I am indebted to this Reddit post and the script that they put together that was created from their research, which let me get my DL360 G6 working. They have a DL380 but I am pleased it works on my system. I don’t have a Reddit account, but show the original author some love if you do! https://old.reddit.com/r/homelab/comments/iw5cew/proxmox_i_created_a_script_to_fix_gpu_pass/

I’ve quoted their script below just in-case the original pastebin link in their Reddit post goes down.

#!/bin/bash
 
#Proxmox 6.2 over Ubuntu Focal Fossa patch for IOMMU.
#this can be done with Proxmox vanilla, just run the script!
 
#based heavily on
#https://forum.proxmox.com/threads/help-with-pci-passthrough.23980/
 
#move to working dir
cd "${0%/*}"
 
#Proxmox 4.4 kernel modding guide dependencies (see the above link)
#apt-get install git screen fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev libperl-dev asciidoc xmlto gnupg gnupg2
#Debian kernel build dependencies
#apt-get install build-essential linux-source bc kmod cpio flex cpio libncurses5-dev
#zfsonlinux has hidden dependencies
#apt-get install dh-python python3-cffi python3-setuptools python3-sphinx python3-all-dev
#even more dependencies
#apt-get install libdw-dev libiberty-dev libnuma-dev libslang2-dev lz4
 
#full list of dependencies, though it may contain duplicates
apt-get install git screen fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev libperl-dev asciidoc xmlto gnupg gnupg2 build-essential linux-source bc kmod cpio flex cpio libncurses5-dev dh-python python3-cffi python3-setuptools python3-sphinx python3-all-dev libdw-dev libiberty-dev libnuma-dev libslang2-dev lz4
 
#rm -r ./tmp/proliant-iommu-patch/
#TODO Only if the user says so on finishing compilation - Ubuntu-focal file is weird so we have to remove
#rm -r ./pve-kernel
 
#mkdir ./tmp/proliant-iommu-patch
#cd ./tmp/proliant-iommu-patch
 
#/tmp/proliant-iommu-patch/pve-kernel/patches/kernel/
git clone git://git.proxmox.com/git/pve-kernel.git
#mkdir ./pve-kernel
cd ./pve-kernel
 
#use a proxmox makefile bug to only get the submodules. THIS iS A BAD IDEA
make -j 2
 
cd ./submodules
 
#saves the hassle of Proxmox using git on its own
#git clone git://kernel.ubuntu.com/ubuntu/ubuntu-focal.git/
 
#TODO UBUNTU_NAME= "$(ls | grep -m 1 ubuntu)" > used for compatibility
 
UBUNTU_NAME="$(ls | grep -m 1 ubuntu)"
 
echo $UBUNTU_NAME
 
#if [ ! -d ./$UBUNTU_NAME/drivers ]
#then
#    #git clone git://kernel.ubuntu.com/ubuntu/ubuntu-focal.git
#    #git clone git://git.proxmox.com/git/mirror_$UBUNTU_NAME-kernel
#    git clone https://github.com/torvalds/linux.git
#    echo ""
#    #git clone $REPOSRC $LOCALREPO
#else
#    echo $UBUNTU_NAME + " is already cloned, skipping!"
#    #cd $LOCALREPO
#    #git pull $REPOSRC
#fi
#
#mv -T ./mirror_$UBUNTU_NAME-kernel ./$UBUNTU_NAME
 
sed -i.new 's/device_is_rmrr_locked(dev)/false/' ./$UBUNTU_NAME/drivers/iommu/intel-iommu.c
 
#cat ./iommu/intel-iommu.c | grep 'device_is_rmrr_locked'
 
#named according to proxmox tutorial :)
#diff -u ./iommu/intel-iommu.c ./ubuntu-focal/drivers/iommu/intel-iommu.c > ./remove_mbrr_check.patch
 
#move back to working dir - TODO this
#cd "${0%/*}"
cd ../
 
echo "building"
make
 
find ./ -name '*.deb' -exec cp -prv '{}' '../' ';'
 
#optional TODO
##dpkg -i *.deb
#update-grub
 
cd ../
 
echo "
 
#find ./ -name '*.deb' -exec dpkg -i '{}' ';'
dpkg -i *.deb
 
update-grub
echo \"=======================================================================\"
echo \"Please reboot your system. Tell u/oezingle if your machine doesn't boot\"
echo \"=======================================================================\"
" > ./install-new-kernel.sh
 
chmod 0777 ./install-new-kernel.sh
 
rm -R ./pve-kernel
 
echo ""
echo ""
echo "===================================================================="
echo "Install all .deb files and update-grub, or run install-new-kernel.sh"
echo "===================================================================="

New Extensions for Email Blocking

Standard

I’ve since added a new amount of file extensions that I would recommend that people running mail-servers also block.

Originally I only blocked a few attachments.

ad[ep]|ba[st]|chm|cmd|com|cpl|crt|eml|exe|hlp|hta|in[fs]|isp|jse?|lnk|md[be]|ms[cipt]|pcd|pif|reg|scr|sct|shs|url|vb[se]|ws[fhc]

However now I’ve added a few more based on suggestions from various sources including extensions that Microsoft recommends to block for users of their Exchange server.

ace|ad[dp]|ani|app|appcontent-ms|appref-ms|as[px]|aspx|ba[st]|cdxml|cer|chm|cmd|cnt|com|cpl|crt|csh|der|diagcab|docm|eml|exe|fxp|gadget|grp|hlp|hpj|ht[acm]|html|in[fsp]|its|jar|jnlp|js|jse?|ksh|lnk|ma[dfgmqrstuvw]|mcf|md[abetwz]|mht|mhtml|ms[chptu]|msh1|msh1xml|msh2|msh2xml|mshxml|msi|ops|osd|pcd|pif|pl|plg|prf|prg|printerexport|ps1|ps1.xml|ps1xml|ps2|ps2xml|psc[12]|psd1|psdm1|psdm1cdxml|pssc|pst|py|py[cowz]|pyzw|reg|sc[frt]|settingcontent-ms|sh[bs]|theme|tmp|udl|url|vb|vb[eps]|vs[stw]|vsmacros|webpnp|website|ws|ws[cfh]|wsb|xbap|xll|xlsm|xml|xn|xnk

It should be relatively easy to copy the above into a regular expression suitable for your mail environment. If you think there’s a way I can optimise this list, please let me know 🙂

Note, this list used to contain the following, but I managed to optimise the expressions and remove duplicates, so the original is as follows (for reference)

ace|ad[dp]|ani|app|asp|aspx|asx|ba[st]|cer|chm|cmd|cnt|com|cpl|crt|csh|der|docm|exe|eml|fxp|gadget|hlp|hpj|ht[ac]|in[fsp]|its|jar|js|jse?|ksh|lnk|mad|maf|mag|mam|maq|mar|mas|mat|mau|mav|maw|md[abetwz]|mht|mhtml|ms[ch]|msh1|msh1xml|msh2|msh2xml|mshxml|msi|msp|mst|ops|osd|pcd|pif|plg|prf|prg|ps1|ps1xml|ps2|ps2xml|psc1|psc2|pst|reg|scf|sc[rt]|sh[bs]|tmp|url|vb|vb[eps]|vsmacros|vss|vst|vsw|ws|ws[cfh]|xml|xlsm|py|py[cowz]|pyzw|ps1|ps1.xml|ps2|ps2xml|psc[12]|psd1|psdm1|cdxml|pssc|appref-ms|udl|wsb|cer|crt|der|jar|jnlp|appcontent-ms|settingcontent-ms|cnt|hpj|website|webpnp|mcf|printerexport|pl|theme|xbap|xll|xnk|msu|diagcab|grp|pst|ps1xml|ps2|ps2xml|psc[12]|psd1|psdm1cdxml|pssc|appref-ms|udl|wsb|xbap|xll|xn

HiFiPi – A High Resolution Audio Player

Standard

This is a quick post to showcase our HiFiPi. I’ll try and update it in the future, but knowing me I’ll probably forget.

The software I used is called Volumio and it’s available here. They have some additional features available as a subscription service which you might like, but the core software is open source and free. They also sell pre-made high quality devices if you’re not into DIY 🙂

I seem to remember having a couple of issues getting it configured the way I wanted it, but I didn’t document any of it, so that will have to wait until I can figure it out. Overall nothing too taxing though.

Transgender Day Of Remembrance – TDOR 2020

Tealight focused in front with out of focus candles in the background
Standard

It seems 2020 hasn’t been a good year for many people and again we’ve lost a lot of people we care about. None more so than our friends in the transgender community. A community of people who often face disadvantage. Normally I’d write grandiose passages to speak of hope but this year I think many of us are at our operational limits.

It’s been 6 months since a friend of a friend decided that they’d had enough, there was nothing left for them in this world and they took their own life. That moment many of us were frantically trying (but ultimately fruitlessly) to save a life and for a time it showed a sense of cohesion amongst people and friendships seemed to form.

That feeling of community and cohesion faded quickly into obscurity after we made our memorial broadcast as we were told not to talk about it any more, despite having to be quiet for a week afterwards due to instructions from the police. Therefore when we could talk about the events free from any legal implications, we were restricted from doing so and we couldn’t decompress or process what had happened ourselves and it makes me feel like the words echoed on our memorial page are hollow. “You have friends and you are loved.

I don’t feel loved by many either, so I can relate. I am truly sorry that our world is a vapid, emotionally vacant, uncaring place with a disgusting excuse for compassion.

While we may not have really known each other, we ended up embroiled in the entire situation (and very emotionally invested) as all we wanted was to try and help. We didn’t want any personal gain or kudos from helping. One of the reasons we helped (besides genuine human decency) is because it could very easily have been one of us and I like to think that people I didn’t know would want to help too.

I light candles on this day to guide wayward spirits home. I hope you all find your way.

Edited on 22/11/2020Paragraph added, sentence order adjusted slightly, and some words added to improve readability.

Review of PrettyLinks WordPress Plugin.

Standard

Since I appear to have angered the all-knowing wordpress.org spam detector, they wouldn’t allow me to post my review, so I thought I’d post it on my own site.

I really want to like this plugin, but the nag screen is incredibly annoying

The features I need are well covered by the free / lite version of this plugin, however it regularly puts up a window asking me to update to the pro version, only giving me the option of “Yes” or “Maybe Later”, there isn’t an option to say “No thanks, I’m happy with the free version, I have no need of the enhanced functionality.”

It adds lots of links that will redirect you from your admin panel to the developer’s page to buy it, so it is incredibly easy to click the wrong button and be redirected to a site you didn’t intend to. I understand they need to show the pages off to people that might have use of the advanced features, but there should be an option to hide all the none free stuff I don’t need.

Since the code is GPL I could go through and maintain a fork of the plugin with all the stuff removed, but honestly I don’t have the time so will probably try another plugin.

I looked at buying the pro version, but it’s not a one time payment, it would be a yearly subscription, at the lowest price is £50/year, which is just not economical for my hobby site for myself and friends. I’m starting to like pro plugins that are one time payments with a certain amount of timed support available, if you need more support after that pay for it, and you get upgrades for that major version number. Your requirements may differ.

Grandstream GXP1610 Reboot-o-matic

Standard

I wrote a nice little script to fix a problem I’ve been having with my work VOIP phone. It would lose connection but the screen wouldn’t let you know it had. I didn’t notice that it hadn’t been connected for *two* weeks until someone left a voicemail.

The phone did have a built in SSH interface, which had a reboot command, so I tried using ssh -i key admin@phone < commands.txt to feed it a bunch of commands. I had to pipe a text command to ssh because it doesn’t use a ‘real’ shell on the phone, just Grandstream’s proprietary command interface, which doesn’t accept commands directly when called from ssh.

This was okay, it restarted the phone but it would also reset the password back to ‘admin’. Not very secure really… So I looked for a way to reboot it using the web interface. I fired up Firefox’s network debugger and started to reverse engineer how commands were processed.

I worked out what I needed to do to login as an admin, and then issue the reboot command. It worked. Nice.

#!/bin/bash
sid=$(curl -k -s -c /tmp/cookies.txt -d"password=hunter2" https://192.168.1.50/cgi-bin/dologin --referer https://192.168.1.50 | sed -r 's|.*"sid": "([0-9a-z]+)".*|\1|' )
curl -k -s -b /tmp/cookies.txt -d"request=REBOOT&amp;sid=${sid}" https://192.168.1.50/cgi-bin/api-sys_operation --referer https://192.168.1.50
rm /tmp/cookies.txt

Really though, I wanted a better solution, sure I could reboot the phone every day to make sure it’s working but what would be awesome would be if my script could check to see if the phone was connected to my SIP account, and if it wasn’t, or there was some kind of error, it could reboot it or at least tell me there was an error.

So I wrote version 1 of my script and got it working, when the SIP connection isn’t registered, it will restart the phone.

#!/bin/bash
# Grandstream GXP1610 Reboot-o-matic v1
# Authored on 12/10/2020
# by jcx
# https://jcx.life
# Licence: GPLv3 (or at your option, any later version.)
# Usage: gsreboot [IP/Hostname]
sid=$(curl -k -s -c /tmp/cookies.txt -d"password=hunter2" https://${1}/cgi-bin/dologin --referer https://${1} | sed -r 's|.*"sid": "([0-9a-z]+)".*|\1|' )
status=$(curl -k -s -b /tmp/cookies.txt -d"request=vendor_fullname:P35:PAccountRegisteredServer1:PAccountRegistered1" https://${1}/cgi-bin/api.values.get --referer https://${1} | sed -r 's|.*"PAccountRegistered1": "([0-9a-z]+)".*|\1|' )
if [ ${status} != 1 ]
then
echo "Requesting reboot on ${1} ..."
date
curl -k -s -b /tmp/cookies.txt -d"request=REBOOT&amp;sid=${sid}" https://${1}/cgi-bin/api-sys_operation --referer https://${1}
fi
rm /tmp/cookies.txt

Now I just need some way to automate it, which is where cron comes in. Cron will run a command however often you like, so I just set it to every 5 minutes to do a check, and now I won’t miss any more important work phone calls.

*/5 * * * * /usr/local/bin/gsreboot.sh 192.168.1.50

Okay, so the first version of the script I wrote, while it works, it isn’t very elegant. It didn’t really report any error messages and wasn’t user-configurable so I’ve rewritten it (v2!) and now it supports some options, and has more sensible error messages.

#!/bin/bash
#################################
#
# Grandstream GXP1610 Reboot-o-matic v2
#
# Authored on 12/10/2020
# by jcx
# https://jcx.life
#
# Licence: GPLv3 (or at your option, any later version)
#
#################################
# Please edit the password below to be the admin account on your GXP1610.
# I defined it here so that you don't need to use it on the command line.
# You shouldn't need to make any other changes.
#################################
 
password="hunter2"
 
#### Begin Script
 
if [ -z ${1} ]
    then
    echo "----------------"
    echo "gsreboot2: GXP1610 Reboot-o-matic v2 by jcx (https://jcx.life) licenced under GPLv3"
    echo "Usage: gsreboot2.sh [IP/Hostname] [Protocol: http/https] [Ignore Certificate Errors: Y/N]"
    echo "Example: gsreboot2.sh 192.168.1.50 https Y"
    echo "----------------"
    echo "This will connect to the Grandstream phone on 192.168.1.50"
    echo "using https and will ignore any certificate errors."
    echo "Use with cron every 5 minutes, as it takes the phone about 3 minutes to boot."
    echo "Don't forget to change the password at the top of the script!"
    exit
fi
 
if [ -f "/tmp/gsreboot2.txt" ]
    then
    rm "/tmp/gsreboot2.txt"
fi
 
if [ -z ${2} ]
    then 
    proto="http"
fi
if [ "${2}" = "https" ]
    then
    proto="https"
fi
if [ "${2}" = "http" ]
    then
    proto="http"
fi
 
if [ -z ${3} ]
    then 
    certignore=""
fi
if [ "${3}" = "Y" ]
    then 
    certignore="-k "
fi
if [ "${3}" = "N" ]
    then
    certignore=""
fi
 
sid=$(curl ${certignore}-s --connect-timeout 10 -c /tmp/gsreboot2.txt -d"password=${password}" ${proto}://${1}/cgi-bin/dologin --referer ${proto}://${1} | sed -r 's|.*"sid": "([0-9a-z]+)".*|\1|' )
status=$(curl ${certignore}-s --connect-timeout 10 -b /tmp/gsreboot2.txt -d"request=vendor_fullname:P35:PAccountRegisteredServer1:PAccountRegistered1" ${proto}://${1}/cgi-bin/api.values.get --referer ${proto}://${1} | sed -r 's|.*"PAccountRegistered1": "([0-9a-z]+)".*|\1|' )
if [ "${status}" = "0" ]
    then
    echo "VOIP account not registered..."
    echo "Requesting reboot on ${1} ..."
    request=$(curl ${certignore}-s --connect-timeout 10 -b /tmp/gsreboot2.txt -d"request=REBOOT&amp;sid=${sid}" ${proto}://${1}/cgi-bin/api-sys_operation --referer ${proto}://${1} | sed -r 's|.*"body": "([0-9a-z]+)".*|\1|' )
    if [ "${request}" = "savereboot" ]
        then
        echo "Reboot request has been acknowledged."
    fi
    if [ -f "/tmp/gsreboot2.txt" ]
        then
        rm "/tmp/gsreboot2.txt"
    fi
    exit
fi
 
if [ "${status}" != "1" ]
    then
    echo "Error: Cannot determine status of VOIP account."
fi
 
###
# Enable this code if you want output on success... disabled by default because it works with cron.
###
 
#if [ "${status}" = "1" ]
#    then
#    echo "Success! Your VOIP account is active. No reboot required."
#fi
 
if [ -f "/tmp/gsreboot2.txt" ]
    then
    rm "/tmp/gsreboot2.txt"
fi

Its grown from 18 lines of code to 109(!). This isn’t bad, considering before I wrote these scripts, I’d never written anything in shell / bash script before. So I replaced the entry in my crontab to run the new script every 5 minutes.

*/5 * * * * /usr/local/bin/gsreboot2.sh 192.168.1.50 https Y

Below is what it looks like in my email client on my Linux box when the SIP account is not registered and it needs a reboot.

Date: Mon, 12 Oct 2020 00:10:07 +0100
From: Cron Daemon <root@localhost>
To: jcx@localhost
Subject: Cron <jcx@localhost> /usr/local/bin/gsreboot2.sh 192.168.1.50 https Y
VOIP account not registered...
Requesting reboot on 192.168.1.50 ...
Reboot request has been acknowledged.

If the script encounters an error, it will email an error response. This looks like the following:

Date: Mon, 12 Oct 2020 00:35:07 +0100
From: Cron Daemon <root@localhost>
To: jcx@localhost
Subject: Cron <jcx@localhost> /usr/local/bin/gsreboot2.sh 192.168.1.50 https Y
Error: Cannot determine status of VOIP account.

I hope this proves useful to you, it certainly has to me. Not only because my phone will always be connected but I also learnt how to do some basic shell scripting. Have a great day!

Setting Up Auto Mounting Encrypted Raid Disks

Standard

This is a little guide (currently under construction) for how I handle encrypted disks on Linux. This won’t be the ultimate ‘tin foil hat’ guide, as the attack vector this is intended to protect from is physical theft of the hardware, so that the data can’t be accessed from elsewhere. It obviously will not handle a targeted hacking attempt or the $5 wrench method, but I believe it gives security and convenience to a level appropriate for me.

xkcd 538: describing the $5 wrench method of breaking security.

The reason this started is because my physical health is deteriorating and getting up to enter a password at the console on every reboot is tiresome. Therefore I came up with a new way of handling encrypted drives to not only increase security but also make things a bit more convenient.

Of course before following any of these instructions, you should be aware of my standard disclaimer.

Caution – You need to secure the location of where you store your key. If you fail to secure your key with an appropriate mechanism, this entire exercise is fruitless.

Examples include: IP restricting access to your key provisioning system, using a strong username and password, using an easy to revoke token based storage mechanism, verifying HTTPs transfer certificates and countless others.

Included below is a method similar to what I use to secure where I store my keys.

Create a keyfile

dd bs=256 count=1 if=/dev/random | base64 > data-keyfile

Upload the keyfile somewhere, for example a HTTPS server with a valid certificate, or S3 or Azure key storage, and then make a script to download the key from where you put it. If you’re storing your key on a HTTPS server, here is an example htaccess file to secure access to the directory to specific IPs and a user/password section to further increase security. This works with Apache 2.4 but the syntax may be different for later versions.

order deny,allow
deny from all
allow from 192.168.1.100

Options -Indexes
AuthType Basic
AuthName "Restricted Access"
AuthUserFile "/secure/path/to/htpasswd"
Require valid-user

Once you have uploaded it somewhere don’t forget to delete the original source file securely from your system (for example with shred).

#!/bin/sh
set -e
# Request the file from somewhere, maybe blob storage, asure, S3 or HTTPS Server, then pipe it through `base64 -d` to decode it from base64
curl -s --basic --user username:password "https://example.net/data-keyfile" | base64 -d

Then move the script somewhere and give it the right permissions

# Ensure the owner of this file is "root"
chown root:root /etc/luks/get-key.sh
# Allow only the owner (root) to read and execute the script
chmod 0500 /etc/luks/get-key.sh

Create the raid

# if all drives are already blank and ready to be added. Replace drives as appropriate.
mdadm --create /dev/md2 -l 1 -n 2 /dev/sdc1 /dev/sdd1
# if you need to create a 'degraded' array with a drive missing.
mdadm --create /dev/md2 -l 1 -n 2 /dev/sdc1 missing

Then encrypt the array

# Encrypt the disk
# Replace md2 with the correct array!
/etc/luks/get-key.sh | cryptsetup -d - -v luksFormat /dev/md2

# Open the encrypted volume, with the name "data"
# Replace md2 with the correct array!
/etc/luks/get-key.sh | cryptsetup -d - -v luksOpen /dev/md2 data

# Create a filesystem on the encrypted volume
mkfs.ext4 -F /dev/mapper/data

# Close the encrypted volume
cryptsetup -v luksClose data

Find the encrypted partitions UUID

$ lsblk --fs
NAME    FSTYPE      LABEL           UUID
[..]
sdc
└─sdc1         linux_raid_mem server:1 a38cbabe-0f12-3643-f3232-998822c5d42
  └─md2        crypto_LUKS             a17db19d-5037-4cbb-b50b-c85e3e074864 

Then create a script to run on boot to automount

#!/bin/bash
if [ -b "/dev/mapper/data" ]
	then
		if [[ $(findmnt -M "/disks/data") ]]; then
		:
		else
    		echo "Not mounted but unlocked... trying to mount..."
	mount -t ext4 -o errors=remount-ro /dev/mapper/data /disks/data
		fi
	else
		curl -s --basic --user username:password "https://example.net/data-keyfile" | base64 -d | /sbin/cryptsetup -d - -v luksOpen /dev/disk/by-uuid/a17db19d-5037-4cbb-b50b-c85e3e074864 data
		mount -t ext4 -o errors=remount-ro /dev/mapper/data /disks/data
fi

if [[ $(findmnt -M "/disks/data") ]]; then
# Anything you want to run after the disks are mounted
		echo "All disks mounted, starting services..."
		echo "Starting samba..."
		systemctl start smbd
fi

and add it to root’s crontab on reboot.

# m h  dom mon dow   command
@reboot sleep 30 && /etc/luks/start-crypto.sh

Don’t forget to disable any services you don’t want to run until the encrypted drives are mounted, for example samba

systemctl disable smbd

Create the mount point

mkdir /disks/data

And finally a script to stop encrypted drives (if required)

#!/bin/bash
echo "Stopping Samba..."
systemctl stop smbd

if [[ $(findmnt -M  "/disks/data") ]]; then
    echo "/disks/data is mounted, trying to unmount..."
	umount /dev/mapper/data
    echo "Attempting to close luks on /dev/mapper/data ..."
	if [ -b /dev/mapper/data ]
		then
		/sbin/cryptsetup -d - -v luksClose data
	fi
else
	if [ -b /dev/mapper/data ]
	then
    	echo "/disks/data is not mounted, but is unlocked, will attempt to close ...."
	/sbin/cryptsetup -d - -v luksClose data
	else
	echo "/disks/data is not unlocked or mounted, nothing to do."
	fi
fi

This work was inspired by an article on https://withblue.ink/2020/01/19/auto-mounting-encrypted-drives-with-a-remote-key-on-linux.html by Alessandro Segala and adapted/changed to meet my requirements.