28 Jan BI Developer (Python + Web Scraping)
Languages: English (must have)
Experience: iGaming affiliate business
Growth Leads is looking for a skilled BI Developer with focus on Web Scraping using Python to join our team. This role will be responsible for designing, implementing, and maintaining web scraping processes to gather data from various online sources efficiently and accurately. You will play a crucial role in collecting data for business analysis purposes, competition analysis, and other business intelligence purposes.
- Data Source Identification: Identify relevant websites and online sources from which data needs to be scraped. Collaborate with the team to understand data requirements and objectives.
- Data Cleansing and Validation: Cleanse and validate the collected data to eliminate errors, inconsistencies, and duplicates. Ensure data integrity and accuracy throughout the process.
- Monitoring and Maintenance: Continuously monitor and maintain the web scraping processes. Address any issues that arise due to website changes, data format modifications, or anti-scraping mechanisms.
- Scalability and Performance: Optimize web scraping procedures for efficiency and scalability, especially when dealing with a large volume of data or multiple data sources.
- Documentation: Maintain detailed documentation of web scraping processes, data sources, and methodologies. Create clear and concise instructions for others to follow.
- Collaboration: Collaborate with other teams such as data analysts, developers, and business stakeholders to understand data requirements and deliver insights effectively.
- Security: Implement security measures to ensure the confidentiality and protection of sensitive data throughout the scraping process.
- Compliance and Legal Considerations: Stay up-to-date with legal and ethical considerations related to web scraping, including website terms of service, copyright, and privacy regulations.
- Proven experience as a Web Scraping Specialist or similar role, with a track record of successful web scraping projects.
- Expertise in handling dynamic content, user-agent rotation, CAPTCHAs, rate limits, and utilization of proxy services.
- Proficiency in programming languages commonly used for web scraping, primarily Python, but also BeautifulSoup, Scrapy, or Selenium.
- Strong knowledge of HTML, CSS, XPath, and other web technologies relevant to web scraping.
- Knowledge and experience in best of class data storage and retrieval of large volume of scraped data.
- Understanding of web scraping best practices, including handling dynamic content, user-agent rotation, and IP address management.
- Attention to detail and the ability to handle and process large volumes of data accurately.
- Familiarity with data cleansing techniques and data validation processes.
- Good communication skills and the ability to collaborate effectively with cross-functional teams.
- Knowledge of web scraping ethics, legal considerations, and compliance with website terms of service.
- Strong problem-solving skills and the ability to adapt to changing web environments
- Experience with cloud-based solutions and distributed web scraping systems.
- Familiarity with APIs and data extraction from non-public sources.
- Understanding of various data formats such as JSON, XML, CSV, etc.
- Plus if having worked with Google docs integration
- Experience with version control systems like Git.
- Strong understanding of HTTP Requests, HTML markup, AJAX
- Experience with Database, PostgreSQL as well Data warehousing
- Benefit if having worked with Playwright or Puppeteer for browser automation
- Benefit if having experience with Amazon Web Services
- Benefit having an understanding or experience in affiliation marketing
COMPENSATION & BENEFITS
We offer you a competitive salary and private health care benefits. At Growth Leads you profit from flexible work schedules, a vibrant work environment and excellent opportunities to learn and grow.
If you feel you have what it takes and that you are just the right person for our team, please send us the below documents to [email protected]
- a letter of motivation
- a curriculum vitae
We look forward to receiving your application!Apply now: [email protected]!