This release reverts a filesystem purge lookup change and adds a script for administrative tasks.
This patch release partially reverts a change to the filesystem backend's purge lookup, adds a script for administrative tasks, catches JSON errors when malformed pastes get uploaded and includes updated libraries for GCS and S3 backends.
Release 1.5.0 contained a simplification to the filesystem backend's purge logic, which could lead to very resource intensive purge cycles on instances with a large storage footprint. This release retains the glob pattern, but re-adds the limited and randomized lookup. The limit gives up searching after 10 times the purge batch size and the randomization prevents re-opening the same, non-expired pastes over and over. Without these mechanisms, eventually all expired pastes got purged, and every further purge reads all pastes from disk, without finding anything, which wastes time and resources.
PrivateBin now adds another script to help perform administrative tasks. The new script, called administration, helps with deleting pastes, removing empty directories, when using the filesystem backend, purging all expired pastes at once and gathering statistics on an instance.
Deleting pastes is relatively easy to do manually with the filesystem and database backends, but more difficult on GCS or S3 storage. The new tool works the same, regardless of backend, letting server administrators delete pastes by ID.
When using the filesystem backend, the purge does not remove empty directories, as they can be reused by new pastes with IDs starting on the same first 2 bytes. These empty directories can now be removed, if desired.
The administration script also can issue a full purge cycle. Instances could disable the automatic purge on paste/comment creation in their configuration and instead use a cron job to run full purges on a schedule. Or it could be issued before a backup, to avoid archiving expired data.
Finally, the script can be used to gather and display statistical information. This includes the total number of pastes the instance hosts, as well as how many of these are expired, of the burn-after-reading type, include discussions and what formatting they use (plain text, source code or markdown).
The release includes smaller improvements to catch a JSON parsing exception when malformed pastes get uploaded to the API and updates the suggested library versions for GCS and S3 storage backends. We had gotten reports of the S3 library in 1.5.0 having emitted deprecation warnings on PHP 8.1.
Benefits of switching to the new release
We recommend to upgrade 1.5.0 instances using the (default) filesystem storage backend as well as instances using S3 storage and PHP > 8.
Update procedure
As usual, you can download the archive for a manual upgrade and can find more details in the installation instructions.
We also offer a Docker container that includes the recommended secure setup with the non-essential files and data outside of the web servers document root.
Changes since version 1.5.0
- ADDED: script for administrative tasks: deleting pastes (#274), removing empty directories (#277), purging expired pastes (#276) & statistics (#319)
- FIXED: Revert Filesystem purge to limited and randomized lookup (#1030)
- FIXED: Catch JSON decode errors when invalid data gets sent to the API (#1030)
- FIXED: Support sorting v1 format in mixed version comments in Filesystem backend (#1030)
Help wanted & greatly appreciated
Apart from the large tasks that require deeper insight and time, there are also smaller issues were help is wanted, topics open to debate and of course many languages that still remain to be translated. We are also still looking for additional long term maintainers among our frequent issue helpers.
If you are interested in helping with any of these points, we have prepared a development guide including design goals, code structure and tools that should get you started. For any questions, you can also chat with the maintainers in the discussion area.
Plans for future releases
The next regular release will focus on user interface improvements.