Syncing again Mac, iPhone, iPad

[john cesta]

So I enter a record into my MAC. It includes simple stuff like name, start date, time and endgame. Nothing special.

It finally syncs to my iPhone after about 15 minutes. Perfect. So I look on my iPad. Nothing.

So I look back on my iPhone, the data in the record was gone.

I look back on the Mac, the data was missing from there too.

It seems as though the iPad did a sync with the empty record and then the iPhone and Mac picked up the missing data and it was gone too.

Any guesses?

Can’t use Tapforms under these conditions.

[john cesta]

I don’t know. Perhaps I expect the syncing to be faster. Maybe I’m impatient.

I was under the impression that if I was in the house with my phone and making entries that 15 minutes latter I could pick up my mac or tablet and work from there and see those entries on that device. But it doesn’t always work that way.

What is the best process when owning 4 devices?

I have Mac, Iphone, ipad and ipad mini.

JC

[David Oxtoby]

Hi, i’ve struggled on and off for a while with the sync across devices when using iCloud or Cloudant, but the best solution that seems to work well is to setup a CloudDB on my Mac and have TapForm devices sync with that. So far no lost data.

[john cesta]

I’m not so sure I am technically qualified to do this.

[john cesta]

It seems as though my iPhone and iPadmini and Mac seem to sync together.

I removed tapforms from my iPad. I know it doesn’t make sense but this is what I’ve found.

[Daniel Leu]

I run CouchDB on my local file server without any problems. All my devices synchronize with this machine.

[john cesta]

Yea, thanks but I am not technically proficient to configure this type of system.

I always thought tap forms would work sufficiently for a small one man operation as myself for syncing a few devices without needing to resort to configuring something like couchDB.

I am happy for those that have the technical expertise and time to do this but I don’t.

I don’t have a local file server. I got away from all that when I left unix and Novell and Citrix.

I’m just s simple guy with simple needs.

I can write my own Tapform programs but not all the other stuff.

Thanks again,

[Brendan]

Hi John,

Apache CouchDB is not really that difficult to configure at the most basic level. It ends up just being a double-clickable application just like any other app on your Mac.

But, have you tried using the Nearby Sync option? That’s very easy to configure too.

I’m sorry for the sync problems you’re currently having. I am working on a longer-term project that should improve the sync story for Tap Forms and give you other sync options.

Thanks,

Brendan

[Sam Moffatt]

I ran CouchDB on my Mac for the longest time, I eventually shifted it to a Linux box that I do some other database work on. I’ve not had issues with sync, even with attachment heavy documents, for a long time. I have a 32GB document and 11GB that I sync fine via CouchDB to multiple devices. My biggest problem is sometimes my CouchDB server is offline and the sync system complains at me that it’s unavailable.

[john cesta]

Yesterday on the Mac I deleted a form with 2400 records and then imported 2400 more. Almost instantly they all synced over to my iPhone and iPad mini. Seamlessly.

Then later that day I added a few meetings into the iPhone. 4 hours later neither of those entries were synced with the Mac or the mini. I rebooted the phone and those entries synced to the mini but not the Mac.

This morning they made their way to the Mac.

It was as if they were stuck somewhere.

Brendan when I read this page it seems to be more than double clicking on an app on the Mac

https://www.tapforms.com/help-mac/5/en/topic/couchdb-sync

Thanks

[Brendan]

John, ya there’s some post installation setup stuff that you could do, but some of it is optional and not necessary if you’re running CouchDB on your own local network.

[john cesta]

It’s all good. It seems to be syncing with all devices. I spent a little time airing the document to all devices and learned a little patience.

Tapforms is awesome.

[Sam Moffatt]

If you’re dealing with iCloud then you’ve got to wait for it’s request throttling limitations as well as bandwidth/latency constraints on your own connection which means you’ll have to leave Tap Forms active in the foreground so that it can complete a sync. It’s the same for other platforms like CouchDB and Cloudant as well. Even on my local network using CouchDB it can take a minute or two for my iPad to fully sync when I’ve made significant changes.

I used to use the iCloud sync on Tap Forms 3 and remember leaving my iPad on power for hours during sync at times to get everything up to date. iCloud is slow and when you can’t cheat by having fully background processes like Apple does, there isn’t any good workarounds for applications beyond keeping them open for a while.

[Pyromixer]

I’m sorry for the sync problems you’re currently having. I am working on a longer-term project that should improve the sync story for Tap Forms and give you other sync options.

I really hope so. I would like to do something with tapforms again, but the sync problems are too big. And too complicated to set up.

But I’ve been waiting for over 2 years: – /

[Bruce O’Connor]

And I again add my name to this list of disaffected sync-ers. None of three suggested mechanisms (iCloud, Cloudant, or Nearby) work for me – either sync doesn’t happen or it is incomplete or wipes out records on one of my 4 devices (2 iPads, an iPhone, and a MacBook Pro). Brendan, you told me a few months ago in a private email exchange that sync was the bane of your existence. Until you get it fixed reliably (Cloudant was good before IBM took it over and it became incredibly complex to setup), I can no longer use your software even tho I have been a user ever since Bento disappeared.

[Pyromixer]

Maybe back to the (Bento) roots with an all-new-database. Less details and features for stable sync and data security.

[Sam Moffatt]

Bento only ever supported syncing from a Mac to an iPad and it was manual sync as well. Though you’re going between two devices and you’re cognisant of having to do the sync otherwise your changes didn’t get to the other side. You had to be aware of it and explicitly do it though perhaps for some folk that would make it more explicit if the same behaviour occurred with Tap Forms.

Multi master sync is hard, especially when you factor in sync failures, devices that might be offline frequently and conflicting edits. My understanding is that Tap Forms automatically resolves the conflicts based on the last record edited. However that can still inadvertently lead to data loss if you edit a record on a device, realise that it’s out of sync and then trigger the sync you will loose data. Last write wins is a common resolution strategy. An alternative would be having to manually reconcile every change and perhaps that’s an improvement but that could in some situations lead to hundreds or thousands of records being out of sync (an updated calculation or script could easily trigger changes to records).

iCloud Sync seems consistently problematic which in part makes sense because it’s backed by Apple’s notoriously finicky platform. Even Apple’s own apps mess up iCloud so I’m not surprised that Tap Forms has some challenges with the platform too. I am more curious about the Nearby Sync and Cloudant failures though.

For Nearby Sync I found quirks at times when everything talked to everyone because you’re basically inviting a race condition to occur. I did have some weirdness at times in propagation that generally got solved by moving it to a star topology where Tap Forms on my iMac had every other device added to it and then every other device only had the iMac. Nearby Sync works on a pull basis when the app is open and only when you’re on the same network, so it is actually easy to open up the app, make changes and realise that your device is out of sync. Nearby Sync I think is great if your devices are at a single location and you can have one instance of Tap Forms (I use a Mac but you could use any device that runs Tap Forms) open consistently for everything to sync against. If you have multiple devices and they aren’t always running Tap Forms when the changes are made then you will end up with situations where devices don’t have all of the changes because Tap Forms needs to be on at the same time on those devices for nearby sync to work.

The Cloudant/CouchDB side should make that better because it’s a hosted server model. Instead of having one Tap Forms app open all of the time, there is a server that is always on. That said if you lose connectivity to that server, you can’t push the changes but offline editing will still happen. There are a few situations where I believe Tap Forms will pause sync indefinitely depending on the error it receives (e.g. unable to connect to the server), that could trip someone up who is expecting it to be working but not realising it hasn’t been. This is generally easy to triage: on the Mac if you look at the bottom right after you make a change the “last uploaded” time should update and if you make a change on another device last downloaded should change; those same timestamps exist on iOS on the Tools tab.

Hopefully that helps folk out and make sync a little more reliable.

[Brendan]

Thanks for the detailed description of the sync process Sam. Sam is exactly right in describing the challenges and the way sync works.

Sometimes things do get out of sync and one way to rectify that is to use the Send Document function to re-send another copy of the document over to your other devices. That assumes of course that you have a device which is the source of truth. It’s always best to have a single source of truth, such as Sam described with the star topology. The Mac is the source of truth and all the other devices sync to it.

If you don’t want to send the entire database over, you can also use the Export Tap Forms Archive command to generate a .tfarc file which you can import into another device. In that scenario, if there are changes or additions made to the form, records, fields, etc. then when you import, Tap Forms will apply those changes. It won’t delete anything though when importing a .tfarc file.

Sync is definitely a difficult thing to understand and to program. That’s why I initially chose CouchbaseLite as a database engine for Tap Forms 5. It contained within it the ability to sync using the CouchDB protocol and peer-to-peer syncing. So Tap Forms relies on that library to provide the syncing for those services. Cloudant also falls in that. Yes, IBM made it more difficult to setup, but it is possible and it does work, as long as you stay within their restrictions, such as records being less than 10 MB big (including attachments) and not storing more than 1 GB of data there.

And now Apple has thrown more curves at us developers by introducing the new Apple Silicon Macs and all the changes in Big Sur and iOS 14. So I’ve been focusing on making sure that I can get Tap Forms to run natively on Apple Silicon. I’m also investigating a new third-party sync engine framework which will hopefully allow me to bring back Dropbox syncing (as well as iCloud syncing) and possibly another sync service or two.

But this is a really big project for me, so it will take a while to build, then test, then fix, then test, and so on… until it’s ready.

Thanks,

Brendan

[Author]

Posts