Issues with importing a large .csv database

[Clint Mitchell]

Hi guys,

I have a large .xls database that I am trying to import into Tap Forms. I have exported the .xls file to .csv. When I check the .CSV file everything has exported fine. The database has 10,937 entries and all of these are visible on the .csv file. However when I import the .csv file into tap forms only 452 of the entries import. All of the others seem to be missing…

The 452 imported entries look fine in Tap Forms it’s just they are the only one’s that imported out of the 10,937 original entries in the .csv database.

Is there a limit to the number of records you can have in a tap forms “form”?

I can’t work out why the other records are failing to show up?

Any ideas? Thanks. :)

[Clint Mitchell]

Attached is the Tap Forms import window when I try to import the database. As you can see at the bottom left it is only attempting to import 451 records.

Also attached is a screen grab of the end section of the same .csv file I am trying to import. It shows that the whole 10,937 records are present in the .csv file, but only the first 451 are importing into Tap Forms…

Attachments:

You must be logged in to view attached files.

[Brendan]

Hi Clint,

Excel typically doesn’t export a CSV file with UTF-8 File Encoding. So I would suggest changing the File Encoding setting in Tap Forms to MacOS Roman which is what Mac Excel uses for saving a CSV file.

That’ll solve your problem.

If the CSV file comes from Windows, then use Windows Latin 1.

[Clint Mitchell]

Hi Brendan,

Thank you! That solved the problem with the number of records being imported. However, with both MacOS Roman and Windows Latin 1 it converts the German text so that certain characters with umlauts are replaced with a different character. Being a database where the data is of German origin it’s kind of imperative that the exact German text is imported as it should be. Please see the below attachments.

Not sure if it helps but I received the .xls file from a Windows user. I then opened it in Mac OS Pages app to export the .csv file as I do not have Excel myself…

Any ideas? :)

Attachments:

You must be logged in to view attached files.

[Brendan]

Pages doesn’t normally support XLS files that I know of. Perhaps Numbers?

Well it is definitely a file encoding issue.

If you want to email me the file I can re-encode it properly for you.

[Brendan]

I mean the original XLS file.

[Clint Mitchell]

Hi Brendan,

Yes sorry, I meant Apple Numbers app.

Is the conversion something I could do with whatever apps I have at my disposal? (Numbers, Libra Office…)

It’s just that I have 11 (minimum) separate databases that I have to convert into Tap Forms and I don’t want you to spend too much of your own time on this as I am sure you are a busy man looking after all us Tap Formers… The largest file is 67.4mb so I’ll likely have to upload to my cloud and email you links.

Thanks for helping with this issue of mine! :)

[Brendan]

Hi Clint,

I don’t mind helping you out with the conversion.

Or if you would like to send me a smaller sample file that has the character issue, then I can figure that out and tell you what I did to get it to import properly.

I also often use TextWrangler (or BBEdit) to determine file encodings and to change them to something I need if possible.

[Clint Mitchell]

Hi Brendan, thank you!

What is your email address so I can forward a smaller file?

I have TextWrangler and am sure I could follow your advice on how to alter the other databases so I can get them importing properly myself.

Thanks! :)

[Brendan]

support@tapforms.com is where you can send any email requests to.

What you can do is just open the CSV file you’ve got in TextWrangler and see if all the characters look proper. Take a look at the bottom-left of the window and it will tell you what it thinks the file encoding is. It doesn’t always get it right. If you see that your umlauts and other characters look wrong, then use the Reopen Using Encoding command and pick a different file encoding. Try and pick one that makes sense. If everything looks ok with all the accented characters, then see if that particular encoding is available in Tap Forms. If not, then what you can do is change the encoding in TextWrangler and re-save the file.

At the bottom of TextWrangler you can click on the encoding name and choose a new one. I would recommend UTF-8 because that can handle all of the accented characters for pretty much all languages.

Save the file and then import it into Tap Forms. Make sure you then select UTF-8 as the encoding in Tap Forms.

That should import all the records.

[Clint Mitchell]

Hi Brendan,

Thanks. That solved the issue. I opened the .csv files into TextWrangler and then resaved as UTF-8 .csv and all of the entries imported fine into Tap Forms.

Thanks for your help! :)

[Author]

Posts