The recommended file size for Auto Insights is up to 10GB. For our typical use cases, like procurement, finance, sales, complaints and workforce analytics, we find that the data can vary in size from 10MB to 5GB.
For example, in a 10GB dataset, a procurement function spending between $20 to $50 billion a year would be able to capture 3 to 4 years worth of transactions easily (assuming the data had approximately 20 columns and 80 to 100 million rows).
Where your .csv file is greater than 2GB in size, we recommend zipping the file before uploading to Auto Insights using 7zip, which is free for Windows. It’s absolutely not necessary to zip your file, but it does make the upload process faster!
What if my data is more than 10GB?
If your data is more than 10GB, we suggest you speak to our Business Solutions team for a consultative session where we can explore techniques such as aggregation or other techniques to cater to your needs.
What if I have terabytes (TB) of data?
Common examples we see of terabytes of data are:
The lowest level, most detailed granularity of transactions for all customers in a loyalty program of large supermarket chain over 3 years i.e. Customer 1 buys 1 apple in CBD store at $2 on 18 March 2019
The lowest level, most detailed granularity of transactions of all credit card holders for the Big 4 Australian banks (or similar size) over 2 to 3 years i.e. Customer 1 spends $50 on Uber Eats on 18th March 2019 at 8pm
In this case, our business solution team can offer a consultative session where we can explore techniques such as aggregation or other techniques to cater to your needs.
What if I have petabytes (PB) of data?
The common cases here are:
Youtube 4K videos (all videos globally)
Facebook content (all content globally)
We don’t analyse unstructured data such as Facebook content or videos in their raw format. However, logs or transactions like user behaviour can be uploaded to Auto Insights if the data is in a structured data format, which will significantly reduce the data size. Again, our Business Solutions team can offer a consultative session where we can explore techniques such as aggregation or other techniques to cater to your needs.
If you have any questions, please contact us at email@example.com.