Dropbox Sync

[Patrice Hof]

Hi Brendan
Thank you very much for implementing dropbox sync in TF.
I try to manage dropbox syncing for 2 days, and I’m experiencing some difficulties…
If I understand, DB sync works basically the same way iCloud sync works, but the sync documents are hosted on the DB space rather than iCloud Drive.

Here is what I did:
1) disable iCloud Sync on all devices
2) delete TF Documents & Data on iCloud Drive
3) install TF 3.5 on the 2 mac and 4.0 on iphone and ipad
4) make sure the encryption key is the same on all devices
5) enable dropbox sync on the 1st device (iphone)
6) wait the TF documents are loaded on dropbox (photos) and the TFSync documents are created on dropbox/…/SyncDocuments
7) enable dropbox sync on the further devices

At this stage, the list of known devices contains only the device I’m working on (iphone lists iphone, mac1 lists mac1, etc) and not the other ones.

The folder dropbox/…/SyncDocuments contains only TFSync docs of my 1st device (iphone).
Even if the mac1 displays the time of last sync on mac1, I cannot find in dropbox the TFSyncDevice.tfs nor the TFSyncDocument.tfx for mac1. The same for mac2 and ipad.

As a result, each device seems to sync with itself only, which is not the goal of syncing!!!

What have I done wrong?
Any clue?

Thanks a lot
patrice

[Brendan]

Hi Patrice,

Well it sounds like you’ve covered everything. Please note that syncing will not happen if Tap Forms is not actually running. That’s a bit different than iCloud where syncing of files can happen in the background.

So are you saying that in the Dropbox SyncDocuments folder you only see one TFSyncDevice file?

When you enable Dropbox sync, Tap forms writes a file to the local SyncDocuments folder here:

~/Library/Containers/com.tapforms.mac/Data/Library/Application Support/SyncDocuments

When you launch Tap Forms Mac, it will check the Dropbox version of the SyncDocuments folder to see if there’s anything new, then it will copy the new files over to the Application Support/SyncDocuments folder. It’s from that folder that Tap Forms reads the files and processes them for a sync operation.

I’m assuming that all your devices are connected to the same Dropbox account?

It can also take quite a while to sync everything to Dropbox if you have lots of files. For me when I do my testing from scratch, it can take a couple of hours to sync everything. I have to have Tap Forms running that whole time. But I have lots of files for my testing.

Thanks,

Brendan

[Patrice Hof]

Thanks for the explanations.
What remains mysterious is the invisibility of the devices in TF Mac.
Actually,TF syncs seamlessly between the 2 iOS devices, but the two Mac are still alone :-(

The attached screenshots may bring some clues.

As you can see, the iphone sees all my devices (1), but the iMac isgawen3 doesn’t (2) even if the dropbox SyncDocuments contains the 4 tfs documents (3). The last pic (4) shows the Application Support/SyncDocuments of isgawen3, which contains only its own documents.

Thank for your help.
patrice

Attachments:

You must be logged in to view attached files.

[Brendan]

Well that’s strange. When you startup Tap Forms Mac, it will compare the Dropbox/SyncDocuments folder with the Library/SyncDocuments folder and it will copy the files over to it when there are files in the Dropbox version that aren’t in the Library version. I’m not sure why that’s not happening for you though. Try quitting the Mac version and launching it again. The files should get copied from Dropbox to Library. They get processed when they’re in the Library/SyncDocuments folder.

[Patrice Hof]

I’ve been exploring a bit further:

I manually copied the content of Dropbox/SyncDocuments to Library/SyncDocuments. At this time, the iMac applied the changes made on the iphone.

Then, I added a record on the mac, and synced. What happened:
1) the tfs file got updated in Library/SyncDocuments
2) no new tfx file created in Library/SyncDocuments
3) no change in Dropbox/SyncDocuments

When the Dropbox/SyncDocuments is modified by the iphone, Library/SyncDocuments doesn’t take the changes into account.

I tried to reset sync on the mac. The devices list get empty, but this had no effect neither on Library/SyncDocuments, which doesn’t get empty itself, nor on Dropbox/SyncDocuments. On the contrary, after relaunching TF Mac, the devices list appears exactly as before.

I tried to reset sync on the iphone. The iphone devices list get empty, and the dropbox/SyncDocuments folder on dropbox.com as well. BUT, the dropbox/SyncDocuments on the mac doesn’t: the files remain on the computer, without the dropbox green label (see picture).

Enabling again dropbox sync on the iphone forces the dropbox/SyncDocuments to clear and the new tfs and tfx files for the iphone are created.

But this change doesn’t applied on the Library/SyncDocuments…

Is there a problem with dropbox?

Thanks for help
patrice

Attachments:

You must be logged in to view attached files.

[gzamba]

I have been having problems with dropbox sync myself…
I initiated a sync from my mac and several folders were created in my dropbox folder… The photos, drawings and share folders have all been populated with lots of files (the correct files), but the sync documents folder is empty!

I then tried to locate the Library/SyncDocuments folder on my mac to manually copy the files to see if that fixed the problem, but I cannot locate that…
I have a dual disk setup, with the system located on MacintoshSSD and all the user files located in MacintoshHD. I searched both Library folders and there is no trace of TapForms! can this be true?

[gzamba]

I have located the Library/SyncDocuments folder but the strange thing is that is located deep inside other folders:
Users/George/Library/Containers/com.tapforms.mac/Data/Library/Application Support/SyncDocuments
and it is Empty!

[Brendan]

Yes, that is where that folder is stored when syncing with Dropbox. It would be empty if Dropbox was not syncing for some reason. The files that are in the Dropbox/SyncDocuments folder should automatically be copied to the Library/SyncDocuments folder.

Also, Patrice, I have noticed that the Dropbox client doesn’t always refresh the Finder windows properly. Sometimes temporarily adding a new file into that folder will cause it to refresh properly. I think that’s a Dropbox client issue.

[Patrice Hof]

I don’t use TF on my 2 macs for 3 weeks. I am very disappointed about that.

Dropbox sync perfectly works between ipad and iphone, but TF on both of the macs still doesn’t copy the Dropbox/SyncDocs files into the Library/SyncDocs.

I don’t want to reset sync, because I’m afraid of breaking the last working sync.

Please Brendan, help me resolve all these syncing issues.

Am I alone to experience the problems?

[Brendan]

Are there any errors in the Console application that might give me a clue as to what is happening? As long as all your devices are listed in list of Registered Devices, then all should be ok.

When you launch Tap Forms and you have Dropbox sync enabled, Tap Forms will scan the Dropbox folder for any changes and copy them into he Library/SyncDocs folder.

There may be a clue in the log file though. Launch the Console application and type “Tap Forms” (no quotes) into the search field, then click on any of the values, then command-c to copy, then paste the contents into an email to me.

Thanks,

Brendan

[thinkingcat]

Nope, not the only one having problems with sync & dropbox.

I have Mac, iPhone and iPad.

I did a reset of the sync’ing yesterday evening (20 hours ago). I first re-enabled sync’ing on the Mac. It wrote the Device and the TFSyncDoc files to the dropbox folder. The TFSyncDoc file was timestamped as 19h58, the time at which it started. It remained 0 bytes for the 30min it took for Tap Forms to process the database and write out the actual details. At 20h28, I had 34Mbytes of TFSyncDoc _still_ timestamped 19h58 (as created and last modified. I think that’s odd (bad) behaviour).

Having anticipated that as an issue, though, I did not attempt to re-enable sync’ing on the iPhone or iPad until that file appeared on all the devices (in Dropbox).

At that point, I re-enabled sync’ing on the iPhone. Here’s where it gets… odd. The iPhone Tap Forms claims that sync’ing is enabled and that the iPhone is a registered device. It doesn’t see the Mac. The Mac doesn’t see it. And NOTHING is added to Dropbox sync folder — not on the iPhone, not anywhere.

This morning, I decided to see if it was just an iPhone thing or what (since nothing had changed) and I re-enabled syncing on my iPad. Same stuff — it thinks it is registered, sees nothing else, nothing appears in Dropbox.

Needless to say, there is no sync’ing going on.

An hour ago (10 hours after re-enabling sync on the iPad), a new TFSyncDevice file appeared in Dropbox — timestamped from the morning. Thirty minutes after that, the iPad acknowledged that there is a Mac that is a registered device. I hold out hopes that there is sync’ing going on now — the top bar of the iPad’s display says “21323 rows remaining”.

Still no sign of the iPhone, but I imagine it might show up presently.

What I _assume_ is going on is a lot of background processing of the files in Tap Forms before files get written to Dropbox. And, as I don’t know how to keep Tap Forms in the foreground on the i-devices for anything like “long enough”, it’s taking a day to catch up. There seems to be a difference between the Mac version and the i-versions in terms of when a file gets created in Dropbox, so it’s impossible to tell when “nothing” is happening.

But, if I have to keep everything open and carefully sequence registering my devices in sync’ing, that kind of timeframe is not especially workable more than once — on setup. Standing in the aisle of a store trying to lookup something from my database is not when I want to see “updating rows 22,832”, for eg.

The only thing that is needfully large about my database is the photos (and I *looooooove* that Tap Forms lets me set them up to be scaled down automagically on entry!); there are only about 4,500 records across a dozen forms. I struggle to believe mine is an exceptionally difficult case.

This sync’ing thing is really killing me, especially when it seems completely opaque and mysterious. I am grateful that it doesn’t appear to have lost data yet! But it’s frustrating. And it’s interrupting my steady stream of statements to my husband about how *awesome* Tap Forms is, and the latest feature I’ve uncovered, and how this has gotten to be *way better* than Bento was!

[Brendan]

Thanks for the detailed description of your sync issues Thinking Cat.

One thing to note is that if you have 3 devices, each with a full copy of the database file already, when you reset sync (or set sync up the first time), Tap Forms generates a sync file that contains all of your data. That’s the TFSyncDoc…[0].tfx file you would see. That file would get synced to your other devices and you would then see the notice in the status bar showing you the countdown. That happens only the one time. Other times the syncing would be much quicker because it has already done the big one at the beginning.

Now, if you have 3 devices, each with a full copy of the database, Tap Forms will do that for each device you press the upload to cloud button on. That’s because Tap Forms must merge the data from all 3 copies of the database. It doesn’t know that it’s all the same, but it still needs to process it to find out. So when you’re standing in line at the store, yes, you might see that message, but that’s only the first time. Subsequent syncs should be much smaller and quicker, unless you’ve added a lot of data in between syncs.

Since you have 4500 records, each with possibly a photo or two, it could take Tap Forms a long while to sync those records up to Dropbox. Syncing happens only while Tap Forms is actually running. If you put it in the background, the process stops.

And with so many photos and/or other files in Tap Forms, those files may be getting uploaded to Dropbox before any of your TFSyncDevice or TFSyncDoc files. That may be why they’re coming in considerably later than expected.

As for the 0 bytes thing on the Mac version, that’s just what Dropbox does. I’ve seen a 0 byte file sitting there for a while before and then finally Dropbox finishes syncing the file and then the file size gets updated. Tap Forms would then proceed to process that file once it has finished downloading from Dropbox.

I hope that clears things up a little bit. I know this all isn’t ideal. Believe me, I wish that I had a much better sync system. I’m currently investigating various services to see which one would allow me to write a better sync system that used a central database rather than the TFSyncDoc and TFSyncFile syncing like I’m doing now. I chose that way because I can write all the code on the client side and not have to write any server side code. Plus Dropbox and iCloud are free for most users (depending on their storage size). Other services would charge money for syncing. I’ve been investigating Microsoft’s Azure product which seems pretty good. But there’s a fee for using it and I’d have to pass that cost on to the customer. Not only that, but I have to write a bunch of server side code to manage syncing and that could take a long time. I contacted the guys at DayOne to see how they’re doing their new DayOne sync service and they said they’re using Amazon AWS and it took them 7 months with 3 senior developers working on it full time. Here at Tap Zapp Software, it’s only me. I am Tap Zapp, and so if I were to attempt to replicate what they have, I can imagine it might take me more than a year.

But nevertheless, I’m still investigating to see if there’s a better solution to all this. Believe me, I want a better sync system too. 99.9% of my support requests these days are that people are having trouble syncing. Eventually we usually manage to get it going. Although I wonder if ANY one is able to sync. It works every time for me, so I don’t know what I’m doing differently or how my Mac and iOS devices are setup differently than other people. It’s a very challenging problem to solve. I have had some people say Dropbox sync or iCloud sync is working perfectly for them, so I do know it can happen. It seems that everyone’s got their own unique situation with different database sizes, different numbers of external files (photos, file attachments, etc.) and different networking environments.

I worked with one guy from Australia for a couple of weeks trying very hard to get Dropbox to work for his Macs in the office. It would never work with their main user accounts. But it did work with test user accounts he setup on the same Macs. So is that a Tap Forms problem or a Dropbox problem? It’s hard to say really.

Anyway, I’ve gone on long enough now.

Sorry for the rather lengthy, and probably not very helpful response.

Brendan

[thinkingcat]

Well, let’s see if we can tease some of this apart — I believe:

1/ Some of these issues will persist no matter what sync service is used, because the inherent problem is the processing power of the i-devices.

2/ Some of the solutions may be in getting messages tied more closely to the actual events (e.g., when is Tap Forms actually done processing the sync), or at least getting us users to know what to look for for “completion”.

If “1/” is true, then it suggests you don’t need to waste cycles learning some other sync service that people won’t want to use, anyway. :-)

So, things I’d really like to know at this point are:

+ what is the timestamp that Tap Forms uses to measure whether it has more recent data than the sync file — is it the timestamp of the file, or something internal?

+ how can I keep an i-device app in the foreground long enough for it to process the database file?

+ how can I know if it actually failed (which is what I begin to suspect happened on my iPhone, which still does not have an entry in the TFSyncDevices in dropbox)?

+ And, if it fails, should I tell that particular device to re-set its sync? (That ties to the first question in my list — if it is an internal timestamp, I may have to reset all devices, always, until they all succeed).

BTW, saying “you only have to go through the long sync once” doesn’t really fly, because the reality is we have to do it whenever one of the devices complains about a failed sync. As long as the sync’ing is fragile and our understanding of how to not cross it up is limited, this is an ongoing problem.

Fundamentally, I suspect that rather than pursuing other sync services, it would be better to put energy into breaking up the database files per form, or some such, as you discussed in another thread. I.e., making smaller chunks. Because users will have big data in ways you can’t begin to predict. Pesky users. :-)

Again — I *really* like what you’ve done with Tap Forms, and think it’s gotten to be far better than Bento. So, I’d like to get these issues resolved not just for myself, but so that they can stop being a distraction for _you_!

[Author]

Posts