📝 Second thoughts

10th December 2020

I managed to pull together the reindexing code needed to hide hidden projects just at the end of last night. It's all working, and with a bit of a once-over it should be good to go.

That said, engaging in the complexity of reindexing gave me second thoughts about the whole thing. This is a workaround in the first place to the coupling in the data structure. Change one thing and you need to adjust everything else to maintain consistency. When I started out on this mini-project I hoped for a reasonably quick fix for a bug and to refactor some code and write some specs along the way. We got all those things done, but it revealed along the way the need to rework the data layer.

Database choice

Lachlan spoke of maybe migrating to MongoDB at some point to give us some independence from the Googly overlords. I think that sounds reasonable; I've usually steered clear of noSQL and favoured Postgres for performance, scaling and data integrity, but this could be a chance to try a different approach. This would also give us a chance to rework the data model to solve some of those problems.

Data structure

I realised that the shape of the data in the database is actually much easier to work with than the shape we process it into via LoadData. The database stores everything key-value rather than in index-referenced arrays. That allows data removal without reindexing every reference. The one missing element is ordering, which I think we could accomplish in the JS either through model classes that provide sorting, or by pre-generating a sorted array of IDs for each ordered resource. I'd lean towards the former because it reduces duplication and therefore data maintenance.

Implementation strategy

This would be a prime chance to use the strangler pattern to change data stores and data structures. We could do so like this:

1. Connect to mongo simultaneously with firebase
2. Write a script to copy all data from firebase to mongo (overwriting whatever's in mongo). Now the two databases are synchronised, but whenever we write to firebase they go out of sync and we have to mirror again.
3. Change each place that updates firebase to also write to mongo. Now the two databases should stay in sync.
4. Write a data integrity checker to follow this for good measure.
5. Piece by piece, change the view code to read from mongo rather than firebase.
6. Now none of the view code refers to firebase anymore. Any reference to firebase can now be removed from the codebase.

I wonder about changing the data model and the data store simultaneously. We could certainly migrate to mongo with the same data and then mutate the structure. But this feels like it would actually be messier without gaining us much - we'd need to deal with two mongo databases to separate the two data models anyway, so we might as well create this separation via simultaneous use of firebase/mongo.