Description
Develop, maintain, enhance, robust scraping system for clients in different domain such as fashion stores, cybersecurity, Construction, Real Estate, so on. Develop backend system, aggregator and reporting using Python & Elastic Search. Ensure steady stream of data flows into our client's systems.
Roles and Responsibilities
Lead technical team New crawler development & maintenance Crawl break alerts Requirement gathering, defining scope for all possible usecase Distributed crawling - design, development, automation & workflows Data management - cleansing, normalization, structuring, data integrity, monitoring, quality, scale Event management - failure detection
Technologies: Python, Java, Scrapy
Industry: Fashion