Refactoring is the root of all evil¶

This last month I’ve spent by cleaning up after deciding on a new method of interacting with the FedBOX repositories.

This new method allows us to aggregate items from multiple collections with multiple search queries. This, of course, has the downside of slowing down everything pretty heavily, so there’s a strong incentive of not overdoing it.

The new method has an improved collection deep loading, which allows us to go through all pages until we reach our desired number of items or reach the end.

One thing that this change underscored was again the slowness of our JSON decoding, so I updated the Go-AP ActivityPub package with JSON decoding based on valyala/fastjson. So far the improvements are not blowing my mind but I haven’t profiled everything since the change so there might be some adjustments to do still.

To further mitigate this, we further added the placeholders for a local in memory cache for BrutaLinks to store loaded collections. This has the downside of poor cache invalidation logic, but I’ll work on it more in the future. Currently we just clear the cache every write we do to FedBOX.

The main reason for this large change in logic was that we need to transparently load from services which are not the local FedBOX instance in the case of federated activities.

The move to this new system is not fully completed, the bugs I had to fix were numerous and tedious to fix so not everything got ported to it.

Some improvements were done to the processing package to allow better cache invalidation for collections affected by activities’ side effects.

Misc¶

I have updated the docker images to be built on top of go 1.17 because there should be some theoretical size and run speed improvements. I haven’t checked, so ymmv.

Ive updated to go-chi v5, on both BrutaLinks and FedBOX.