Informacje o stanowisku

On behalf of our client, hybrid iPaaS & AI automation platform, SD Solutions is looking for a Backend Engineer to join our team.

SD Solutions is a staffing company operating globally. Contact us to get more details about the benefits we offer.

This role involves building scalable data extraction services, developing high-performance web scraping solutions, and integrating AI-powered data processing pipelines. You will be responsible for end-to-end development, from designing efficient web crawlers and API integrations to optimizing data pipelines for large-scale processing.

This is an exciting opportunity to work with cutting-edge AI technologies.

Responsibilities

• Develop and maintain scalable web scraping and data extraction services , ensuring efficient and reliable data collection.
• Build and optimize web crawlers using tools like Playwright, Scrapy, or similar technologies.
• Integrate and interact with various APIs (REST, GraphQL) , parsing documentation and extracting relevant data.
• Design and manage data pipelines , ensuring seamless automation and processing of large datasets.
• Utilize vector databases (Pinecone, FAISS, etc.) for efficient data storage and retrieval.
• Troubleshoot and resolve technical challenges related to web scraping, concurrency, and data processing .

Minimum Requirements

• 5+ years of experience in Python (or a relevant language) for data extraction, processing, and asynchronous development using frameworks like asyncio, FastAPI, or similar.
• Experience in web scraping and crawling, using tools like Playwright.
• Familiarity with API structures (REST, GraphQL) and the ability to parse API documentation.
• Experience in building scalable services for data extraction and processing.
• Experience in building high-performance, async services, with a strong understanding of event-driven architecture and concurrency.
• Quick learner with the ability to adapt to new tools, frameworks, and techniques efficiently.
• Strong problem-solving skills and the ability to work independently on complex tasks.
• Previous experience in scraping web data specifically for LLM-based applications - advantage.

Nice to have:

• Experience with document parsing techniques (HTML parsing, NLP for text extraction).
• Familiarity with vector databases (e.g., Pinecone, FAISS)
• Knowledge in data engineering and pipeline automation.
• Knowledge of AI model fine-tuning and prompt engineering
• Experience in web crawling/scraping and work with LLM

About Company

We are a NO-CODE platform that revolutionizes how organizations automate business processes and seamlessly integrate with their core software. Acting as a dynamic bridge , our platform enables businesses to build in-house applications, create powerful workflows, and manage dynamic data tables that integrate effortlessly with ERP and SaaS solutions .

With over 20 years of experience in developing and implementing CRM, ERP, and SaaS interfaces , our team is dedicated to delivering scalable, efficient, and intelligent automation solutions to enhance business operations.

Warszawa - Oferty pracy w okolicznych lokalizacjach