Database Design Data Extraction Automation Web Scraping
Want to scrape information from up to 30 websites to pull information into a database and enable manipulation of the
data to be collated into an online report. We are not software engineers, and there are gaps in our UX, workflow, and the
scalability of the platform.
We are seeking a qualified software engineer to help us build a scalable scraping platform and ensure our data feeds
are tested to be robust. Key skills required for this will be experience in scraping websites, database development that
can be categorised and manipulated by subject matter experts, and the development of high-quality online dashboards & reports.
Deliverables
- Review and critically appraise existing software tools that could be used
- Recommend alternative approaches, providing reasoning for adopting a different approach or tool
- Using a low or no-code software (or similar), scrape the data, including references to documents, into a common database
- Implement error reporting to flag any future issues with the data scraping
- Develop a short ‘how to’ document to support ongoing maintenance of the system.
- Scrap information from the nominated websites
- Support the development of an online platform to share information provided in the database and reports
Preference for experience in web scraping, database development, Power BI, or similar tools.