I am curious how to handle replication with high concurrency. I'm not sure how you detect dangerous reads+writes to the same key (tuples/fields) across different replica machines. In other words, multiple master.
I am aware Google uses truetime and some form of timestamp ordering and detection of interfering timestamps. But I'm not sure how to replicate that.
I began working on an algorithm to synchronize database records, do a sort, then a hash for each row where hash(row) = hash(previous_row.hash + row.data)
Then do a binary search on hashes matching/not matching. This is a synchronization algorithm I'm designing that requires minimal data transfer but multiple round trips.
The binary search would check the end of the data set for hash(replica_a.row[last]) == hash(replica_b.row[last]) then split the hash list in half and check the middle item, this shall tell you which row and which columns are different.
My assumption here is that the replication model described is one leader, multiple replicas - so all writes are applied to the leader and the followers are only used for reads.
This is a pattern commonly used with other databases such as PostgreSQL and MySQL which avoids a huge amount of complexity involved in multi-leader setups.
There's also the hot standby leader configuration that the higher Heroku Postgres tiers do. The standby leader receives writes asynchronously, so during failover you can actually suffer a little permanent data loss. Really should have a bigger warning label.
I only ever do single leader, multi follower like you describe.
I'm not sure about the details about spanner, but one thing still remember - it was much cheaper to read stale data (<10secs ago), as it would've been available from any server, not just the leader, and also that transactions were limited to 20mb or was it 40mb. Although that was 8 years ago, so things might be different now.
I am highly interested in parallelism and high concurrency. I implemented multiversion concurrency control in Java.
https://github.com/samsquire/multiversion-concurrency-contro...
I am curious how to handle replication with high concurrency. I'm not sure how you detect dangerous reads+writes to the same key (tuples/fields) across different replica machines. In other words, multiple master.
I am aware Google uses truetime and some form of timestamp ordering and detection of interfering timestamps. But I'm not sure how to replicate that.
I began working on an algorithm to synchronize database records, do a sort, then a hash for each row where hash(row) = hash(previous_row.hash + row.data)
Then do a binary search on hashes matching/not matching. This is a synchronization algorithm I'm designing that requires minimal data transfer but multiple round trips.
The binary search would check the end of the data set for hash(replica_a.row[last]) == hash(replica_b.row[last]) then split the hash list in half and check the middle item, this shall tell you which row and which columns are different.