Developer for Web Scraping System (Russian speaking)

We are looking for an experienced, self-driven Developer to build, maintain, and optimize a scalable system that will scrape real estate listings from over 10,000 websites. This role requires a deep understanding of web scraping, data processing, and database management to ensure that our system can efficiently extract and store real estate data at a large scale. You’ll have the flexibility to choose the stack that best suits the project needs.

Key Responsibilities:

• System Design & Development: Build and maintain a robust, scalable web scraping system to extract data from 10,000+ real estate websites.

• Data Extraction & Processing: Develop advanced scraping algorithms and handle large amounts of data. Ensure high accuracy in the scraped listings (including property details, images, and other metadata).

• Automation & Scheduling: Implement and optimize automated scraping tasks that run at specified intervals to keep data up-to-date.

• Error Handling & Logging: Create comprehensive error-handling and logging systems to monitor the success of scraping tasks, identify potential issues, and address challenges promptly.

• Data Storage: Design and manage databases to store the scraped data efficiently while ensuring smooth retrieval and processing.

• Compliance: Ensure the scraping process adheres to legal and ethical standards, including respect for website terms of service, robots.txt, and relevant privacy laws.

• Performance Optimization: Continuously improve scraping speed, reliability, and system performance to ensure the system is running efficiently at scale.

Required Skills & Experience:

• Proven experience in web scraping and automation, using modern libraries, tools, and frameworks.

• Expertise in working with large-scale data and databases.

• Solid understanding of API integration, HTTP requests, and handling various web technologies.

• Experience with cloud services for data storage and task automation.

• Knowledge of scraping best practices, including handling rate-limiting, CAPTCHAs, proxies, and anti-scraping measures.

• Experience with task scheduling (e.g., cron jobs, Celery, or similar).

• Familiarity with version control systems such as Git.

• Ability to work independently, troubleshoot, and find innovative solutions to problems.

What We Offer:

• Competitive salary and benefits package.

• Opportunity to work on an exciting, large-scale web scraping project with high impact.

• Flexible working hours and the ability to work remotely.

• Collaborative and innovative work environment.

• Opportunities for growth and learning.

Screening Process:

The screening process may include a small paid task to assess your technical skills and approach to solving real-world challenges.