Slow Synchronisation

[John Mathew]

Hi,

I have set up Cloudant syncing on my machines. However, making the slightest change to a record on one machine initiates “Sync sending” of, at times, hundreds of thousands of records (see attached pic).

Am I doing it wrong?

Attachments:

You must be logged in to view attached files.

[Brendan]

Hi John,

After the first successful sync, Tap Forms will only sync just the changes. Perhaps it hasn’t fully synced the first time?

I would suggest signing out, then signing back in again and see how that goes.

How many records do you have?

Those counts don’t directly correspond to the number of records in your database. They’re actually a sequence number of changes that have been made to the database, including records, forms, fields, etc.

[John Mathew]

Hi Brendan,

I have about 10,000 records in total and I’ve been synced for a couple of weeks now. I would have thought the file would have fully uploaded by now. Should I just be patient?

I have attached a screen shot showing the present situation.

Attachments:

You must be logged in to view attached files.

[Brendan]

One other thing you can try is to start the sync from scratch.

So sign-out of Cloudant from within Tap Forms on all your devices.

Then login to the Cloudant dashboard and delete the database from there.

Then on that document in Tap Forms, go to the Maintenance screen in the Preferences area, click the Compact Database Button. That’ll purge deleted records and old revisions. It will be less data to sync. If you delete something, the deleted record still gets synced.

Then sign-in to Cloudant again and there should be less things to sync. Although the counts may not change, it should hopefully go quicker.

Thanks,

Brendan

[John Mathew]

Thanks Brendan! Just one question…. must I create a new cloudant database before signing in again after the “compact” process?

[Brendan]

No. When you sign-in to Cloudant, Tap Forms automatically creates a database for you. In fact it does that every time you launch Tap Forms and open a document. If the database exists already, the create database command silently fails.

[Sam Moffatt]

One thing to note is that the number at the bottom is an aggregate count of all changes done in TF during your current active session. If you edit the same record multiple times then the number will increase over time. This number includes changes you do to fields, any time you create a link between records and then any edits you make to the forms. Cloudant, and CouchDB, are eventually consistent data stores so you can quit TF and when it comes back it’ll try to figure out what changes are missing and push them back up. The number I think can also be a bit misleading at times because some changes change more than one record which also may result in the number being inflated.

[Brendan]

That’s right. This number you see is actually a sequence number that gets incremented for every change. The number on the right his the last sequence number that was synced and the one on the left is the position it’s currently at. It will eventually reach the largest one and that’s usually when things stop. But I’ve also seen the one on the right increase just as the one on the left catches up with it. It’s a bit strange, but it was the only way I could figure out how to show some sort of syncing progress. Couchbase Lite doesn’t really give a good indication of sync progress.

[Sam Moffatt]

What happens is during the initial handshake it will do a request to get a set of changes however those changes are batched. CBL knows that it has at least one more remote change (the response from bulk revs) to process but it’s busy processing earlier requests. Those earlier requests for individual documents get pushed and the counter increments on both sides: right because there is a new change being pushed and then left eventually when it is confirmed. Once the earlier request is done, CBL moves to the next response to bulk revs changes and starts processing it as well. Eventually it finishes all of the bulk rev changes needed to reconcile state on both sides and it calms down again. Since the bulk revs changes are processed sequentially it results in this weird behaviour where it looks like it’s about to catch up before getting further behind again as it starts to process the pending changes. The number itself is just the number of requests it has scheduled to make (right) and then the number it’s completed (left).

The other thing I’ve seen is that the changes being replicated result in conflict resolution on other devices which then results in changes being pushed back as well. The worst case is a full on replication storm where each replicated change results in new changes that are then replicated.

[Brendan]

I used to have Calculation fields updated automatically when a list of records is fetched. If those calculations included a reference to NOW() or TODAY(), Tap Forms would get into a sync loop. That’s why I added the Refresh Records button at the bottom of the records list view. That refreshes the records one time and then those changes can sync without them triggering another refresh.

[Author]

Posts