I recently upgraded to Tap Forms 5 and am working at getting the sync set up with Cloudant. I have the account set up and have entered my login information in Tap Forms 5 in the sync settings. On the initial sync I get the following error:
“Synchronization has failed – Error sending to Cloudant server.: the request entity is too large”
Has anyone else seen this error or know how to correct it? I see in the dashboard for Cloudant that a new database has been created and it is several MBs in size so I know it has transferred data.
If you have a file attachment that is too big this can happen. I think Cloudant’s limits is around 100 MB. If you have attachments that are that large, consider using the alias function instead (the arrow with the + below it) on the File Attachment field. I think it may also be an issue if you have multiple attachments that add up to more than 100 MB and they’re all trying to be sent at the same time.
Brendan, thanks for your quick response. In looking through all of my records and any file attachments that I have the maximum sized file I have is 4 MB. Do you think there may be a user preference setting in Cloudant that may be limiting this?
As far as the database size and potentially all files together exceeding that 100 MB limit, my entire database size is 72.5 MB (zipped).
I don’t think so. One other possibility could be if you have any Note fields with images in them? Or a really really large note field?
Brendan, the entire database is 72.5 MB in size. My note fields do not get that long and the images I use are generally low-res web sized images.
Hmm… Ok, one thing to try is to sign-out of Cloudant within Tap Forms on each device. Then delete the database in Cloudant. Then sign-in again in Tap Forms on one device only. Tap Forms will re-create the database. Let it finish the sync uploading the data to Cloudant. Hopefully it gets through. If it does, then sign-in on the next device. Wait for it to finish syncing, and so on.
If it still happens, maybe send me a backup copy of the database and I’ll try to find out which record is causing the request too large error.
Brendan,
I finally had a chance to sit down and work with this again. I went ahead and deleted all attachments in my database as that seemed the most likely possibility. That did the trick. Evidently there could have been a corrupt file or something impeding the sync process.
Thanks for you input.
Jon
I am having a similar problem. In my case I was switching from local sync to cloudant. Local sync worked great, but was tiresome across all my platforms, so I thought I give cloudant a try.
As I migrated each of my DB to cloudant, I got message this on each of my databases: 1.2 MB, 78.7 MB, and 710 MB. Eventually this seems to go away with the smaller DBs.
BUT, I still get this on the 710 MB DB. Now there are attachments in this DB that range from about 10-30MB. Oddly, everything is there, attachments included, so I am not quite sure what the message is referring to or means in relationship to the state of the db.
Hi Ars,
It could be that some of the attachments may not have synced to the other devices. Or it could be that Couchbase tried again and it worked eventually. Or some of the attachments synced and then when it tried again, the rest of the attachments synced and didn’t cause the error the next time. Couchbase does have mechanisms inside it to try the sync again if it fails for various reasons.
I spoke too soon. I am getting the size error on all my DBs, large or small almost all the time when I open them. This is true of the iMac, iPhone, and iPad. Also, record updates are inconsistent (meaning sometimes they come through, sometimes not). I am guessing that is becuase of the size error.
Do you have any Note fields with a lot of data in them? May be that’s causing the request to be too large.
Can you try on a new small simple database document to see if you still have this problem? I remember seeing this problem during the time I was originally implementing Cloudant sync, but I haven’t seen it lately. And I have some fairly large databases that I sync.
I tried a couple of experiments:
1) Took the “Sample Database” and added it to Cloundant. All iDevices (iMacs, iPhone, iPad) readily synced.
2) Added a very long note. All iDevices (iMacs, iPhone, iPad) readily synced.
3) Added a large attachment file (~70MB), one I had in another DB. It took some time to upload. It eventually appeared in all my iDevices. The second iMac was first, iPhone second. The iPad seemed to not sync, but with an exit and re-enter of TF, the attachment appeared.
4) Repeated the above, but created a new homework record and added another attachment. It took some time to upload. It eventually appeared in all my iDevices. But the second iMac was dead last this time in having to appear. I did notice that the new record synced almost immediately, but the attachment took its sweet time in appearing. Makes sense since it was about 50MB too.
Results would seem to be inconclusive as to why my other two DBs are throwing up the sync error. They have large attachments, (1-70 MB), but added the same files to the “Sample Database” and did not cause the error. The have MORE attachments, but the size of the DB does not seem to matter much here.
I’ll experiment some more.
OK, I was woking on the larger of the two DBs. All I was doing was adding simple records, no attachments. It was throwing up quite a few of the size error messages both on the machine I was working on and on my other iDevices. Furthermore, the iDevices were not even syncing with the additional records.
Is there an easy way to clone this DB? I want to clone it and remove the forms that have the attachments and see if this makes the errors stop. I do not see an easy way to do that without endangering the current DB.
Actually, never mind. I decided the thing to do was to try and delete the largest attachment and then work my way down to the smaller ones to see if I could isolate the issue. This turned out to be fruitful. I delete the largest, 81.8 MB, and the error continued. Then I deleted the next largest at 62.6 MB and the problem vanished. Furthermore, the other devices updated nearly instantly as I added records.
So, I then added the largest file back in. Sure enough, the error started up again. Deleting it removed the error. It is thus clear, I think, that “large” attachments are an issue.
Do you know if there is a limit to attachment size?
And one more thing. I moved over to the second DB that was having the same issue and tried to fix it in a similar fashion. It was not as successful. The odd thing is that the DB was about 80MB and the largest attachment was about 75MB. For the previous DB, when I delete the attachments the DB size went down as I expected. When I deleted the large attachment for this DB, the size TF reports did not decrease, even after several compactions. It should have been reduced to ~5MB. When I look on Cloudant, it does show that it is ~5MB now, but TF says 79MB.
To compound the mystery, I have a search filter than says “show me the record if the file attachment field is not blank”. The record with the deleted attachment still shows in the search field despite being “empty”. The other DB has a similar setup, but it behaves as expected. When I deleted the attachment, the record no longer shows in the search.
I even went so far as to delete the offending record from this DB (the record with the large attachment) and TF still shows the 79MB size despite compaction.
Hmm… That’s odd. It should not show that large after deleting the attachment and compacting.
If you right-click on your .tapforms document in the Finder and navigate into the “attachments” folder, do you see your file there still?
There are two attachments folders in the .tapforms document. The first is at the same level as the db-xxxx folder (called “Attachments”), the second is inside the db-xxxx folder (called “attachments”).
For both DBs, the “Attachments” is empty. Both show the last modification date of Aug 21, 2016.
For both DBs the “attachments” folder has .blob files, with a recent modification date. Both have some “large” files that are roughly of the size of my attached files
For the DB that is now working, the sizes of the blobs would indicate that the two largest files have indeed been removed. However, there are 49 .blob files and I only have 23 attachment files in the DB. I do not know if that os relevant.
For the DB that is still throwing the error, there is a blob there that is roughly the size of the file I deleted. So I guess the answer is “it is still there”.
Lastly, there is definitely a limit with CLoudant as to the size of the attachment file. I added the largest file I removed from my first DB (which is now working) to the test DB I was experimenting with earlier. Sure enough, I now get the error. It is 81 MB. It will be a pain, but I suppose I can split these files into smaller pieces. Some are video files, some are pdfs.
If we can find a way to fix the second DB to remove that pesky attachment, I can split that file up too.
If that file is no longer referenced in the database you should be able to just delete it. However, normally the Compact function does that for you. Couchbase Lite does keep only one copy of the attachment even if more than one record is referencing it.
That did the trick. I had to delete from the Mac version first when TF was not running. Once I got that squared away, I had to delete the db on the other devices and re-send it via local transfer. The delete was necessary because I’d get the size error despite Cloudant showing that they had been deleted.
Base on what I have seen so far, I am guessing an attachment needs to be less than about 60MB.
This may work better with my CloudKit implementation as I don’t think it would have that same limit. But I haven’t tested it yet with such large attachments.
I had this problem with jpg files in a photo field. The first time, I had to delete all of the records and reimport them. I then imported one photo at a time. The photo was less than 2 meg. It still would not sync. I changed the photo size in the field properties to large and everything now works fine (I think).
Brendan,
I too receive “the request entity it to large” message when trying to move from cloudant.com to IBM Cloud. I do not have any attachments or long note sections. My database is a simple record of church members and visits made. There are 1141 records in the contact form and over 2800 in the visits form. I’ve compacted the database and it is still approx. 200 mb. This database is very important to us. I have four other pastors that sync to it. The cloud sync is essential because we are seldom in the same area during the day to do local sync. Any suggestions on what I can do since the IBM Cloud Bluemix only allows 100mb?
Tim
Hi Tim,
Actually the Lite plan allows you to sync up to 1 GB of data. The issue is each record must be 1 MB or less, that includes photos, file attachments, drawings, etc.
I would suggest contacting support@cloudant.com and ask them to raise the limit. I’m sure they’ll tell you no initially. But maybe if enough Tap Forms customers contact them asking about this they’ll increase the limit just to make us stop asking. :)
Check each of your forms, even forms you might not be using frequently to see if there are any records in them that are too large.
In fact, with the latest version of Tap Forms 5, I automatically take you to the first record which caused the “request too large” error after a sync fails to complete. That way you can inspect the record and look for anything that might put you over the 1 MB limit for that record. Sync…, rinse…, repeat….
Is there a way to check the size of each record without having to sync?
<1 MB attachments and records? Ouch!
Is this just a limit for the Couldant Lite “free” level or do the paid levels have a larger limit?
At the moment no. I do have some debug code in Tap Forms which dumps the record size to the log, but it only works when I’m running within Xcode. And it’s just for the raw content size of the record, without the size of the attachments.
On the Sync error with Cloudant: I ran into this with one of my DBs with attachements. Even when I deleted the attachment, I still got the error whenever I synced. I even tried to delete the offending record and that still did not clear it.
To clear it, I turned off syncing in the offending DB on all my devices. I then deleted the DB from each device except one. I then logged into Cloudant and deleted the DB. I then re-enabled cloudant syncing and then pushed copies of the DB to my other devices. A pain, but it got rid of the error.
I synced again and found the first record. The only thing I could see different was the text font in the notes section was very large and a font I have never heard of. I changed it to a normal font and had it do the same for all records. I synced again and it worked!!! Thank you!
Hi Brendan,
I too am having the file size issue when migrating to the new IBM platform <“the request entity is too large” >. I want to keep using this since I share the db with my wife who has a seperate AppleID so iCloud sync is not appropriate.
Sidebar Suggestion: I don’t know how hard it is or even if it is possible to use Family sharing on iCloud to allow multiple users access to an iCloud synced db?
Anyway, I have a largish number of photos in my db, many of which are taken with an iPhone so are 2-4MB in size which is what is killing the sync. I proved this by manually adjusting the size (to <1MB) of the file at which the sync stopped and retried sync – it worked and then stopped at the next >1MB file.
I emailed IBM support as you suggested yesterday and got a reply that they would consult with the engineering team about lifting the 1MB limit – fingers crossed – I’ll let you know the result (although being IBM I won’t hold my breath!)
I think the Cloudant sync option is worth persevering with because it works really well.
kind regards
Bruce from Aus
Hi Bruce,
Family Sharing isn’t for iCloud syncing. It’s just for sharing the same apps and iTunes music amongst the members of your family for a single purchase. So that won’t help.
Thanks for contacting Cloudant support about this issue. They basically told me the same thing. I think if they get more pressure from customers, then hopefully they’ll increase it. It’s not like it’s a technical limitation of their platform or anything. The older Cloudant infrastructure supported up to 64 MB (so they told me). What they told me now was they set it to 1 MB to improve their service to provide improved performance for all customers syncing to their platform. They can probably have more people syncing simultaneously if smaller requests are coming in. But maybe they’ll see that it’s just way too limiting to enforce such a small request size for so many people.
Of course, if they do increase the 1 MB limit, they may just double it and that’ll still be too small.
As it is, with the previous Cloudant service, I still received support emails from some customers who had multi-hundred megabyte files that the were trying to sync. Or they would have multiple large images or file attachments in the same record which would cause the error too.
Thanks,
Brendan
Hi Brendan,
Well I finally got a response from IBM after bugging them a couple of times. Here it is:
“I’m afraid that 1 M limitation is cluster-wide, the purpose is to reduce the impact of noisy neighbours and ensure a better overall performance.
To get the list of your doc size, you could set up a similar view in your source db as below and query it with arguments “reduce=false&descending=true&limit=100″.
docSize”: {
“map”: “function(doc) {\n\tvar size = toJSON(doc).length;\n emit(size, size);\n}”,
“reduce”: “_stats”
}
Moreover, specifying the “worker_batch_size” in your replication doc to a small number, i.e. 1 might help to replicate other docs less than 1M successfully. Hope this helps.”
Seems they will not reconsider the object size.
I am now proceeding to locate and reduce the size of the offending files (almost all Jpeg files) on my Mac version to overcome the problem. Luckily there are only about 150 of them so it’s manageable if a bit of a pain.
FYI I also researched other CouchDB hosting services but any I found are quite expensive and meant for larger a scale db.
I would be interested if anyone else has found a low cost alternative hosting service (although it’s har to beat the IBM free 1GB (which btw their new website indicates will be perpetually free – if we can trust that claim!)
Thanks again for your help. Great support as always.
Regards
Bruce
Hi Bruce,
One of my other customers has had some success with a low cost CouchDB sync service. He wrote about it in this forum post just the other day:
CouchDB is running on my own 1&1 Cloud
He also just told me today that he got it working properly with an SSL certificate and it’s working well for him now. And I think the hosting provider charges about EUR 4.99 per month. I don’t know what the storage size or throughput is provided with that plan though.
You can also use a Mac (or even another type of computer) in your own network to install Apache CouchDB yourself for free. I have another few customers (that I know of) who are also doing that.
Sorry for my bad language skills…
Yes my little cloud couchdb server at 1&1 is running fine and very fast! Every Tap Forms sync is perfect, fast, instantly. Big files, or smaller files into the database, no problems ’til yet. Very faster than the iCloud Sync!
You need: 1&1 Account, one Cloud Server Package (on the german site the prices starts at 4,99 Euro per month. Diskspace is 30GB SSD on my small S package. And for easier ssl handle you need a domain from 1&1 who is switched to the servers ip adress. You get a certificate from 1&1 for one domain for free.
You can change the serversize everytime if you want. One day bigger, two days smaller, never changes, your choice.
My server at the moment is this one:
You can install couchdb with this package with 1-Click install. After this you must do some small things in a ini.file… certificate files copy on the server etc. Thats all.
You have the choice of more then 100 1-Click install apps also. A shopsystem or a CMS.
Then you can administrate your couchdb database via the couchdb Backend on the webbrowser. You must never put a finger on the serverthings on 1&1. Only pay the invoices from 1&1, thats all :-)
You can give user permissions with individual passwords into this backend. So your employees can work on a Tap Forms database, or your wife or all together. If your fired an employee maybe, you can delete this user from couchdb and he cannot login in the Tap Forms Database again.
The own Mac Server at home
I’m was thinking about this also. But you must check always the hardware, always check if the disk ok inside, if the Mac ok and check all security rules to make the server safe against attacks. Then you need a dynamic DNS service. And here in Germany we have slowly internet connection inbound. Outbound is no problem, 100, 400 Mps but inbound very, very slower. This is a problem too here in my area.
At the end is it a calculation: 200 Euro for an older MacMini, Costs of power and maybe the Mac works for two, three years without problems. Then I need the next mac.
Now I paid 60 or maybe 70 Euro for a year for all connected users, all things together and I can make my jobs without headaches.
.
