Data Scraping Studio
p/data-scraping-studio
Point and click screen scraping tool using CSS selectors
Vikash Rathee
Puppeteer API β€” Convert your website into an API
Featured
6
β€’
Introducing our new integration of Puppeteer with scraping agent to allow users to write custom puppeteer scripts in Node.js backed by Agenty cloud for performance and scaling.
See examples on sidebar or on Github - https://github.com/agenty/agenty-chrome-api
Replies
Vikash Rathee
Hi Product Hunters! I will be live today for discussions, please ask question or share your feedback. I will try to respond all questions and plan the improvements in next version, if you have any feedback.
Chris Messina
Top Hunter
@vikashrathee wait, so how does this turn a website into an API?
Vikash Rathee
@chrismessina Just write a puppeteer script and use that to send POST request to https://chrome.agenty.com/function API to turn the website into API. For example, here is an example code to navigate to site > extract 4 fields > return JSON const fetch = require('node-fetch'); const jsonBody = {"code" : "// Read the `url` from request, goto the page, extract products\n// capture screenshot and return the results\n\nmodule.exports = async ({ page, request }) => {\n const response = await page.goto(request.url);\n console.log(response.status());\n const result = await page.evaluate(() => {\n const data = [];\n var products = document.querySelectorAll('.product_pod');\n for (var product of products) {\n data.push({\n product_name: product.querySelector('h3').textContent,\n product_price: product.querySelector('.price_color').textContent,\n product_availability: product.querySelector('.availability').textContent,\n product_image: \"http://books.toscrape.com\" + product.querySelector('.thumbnail').getAttribute(\"src\"),\n product_link: \"http://books.toscrape.com\" + product.querySelector('h3 > a').getAttribute(\"href\")\n });\n }\n return data;\n });\n\n await page.screenshot({ path: 'books-toscrape.png', fullPage: true });\n\n return {\n data: result,\n type: 'application/json'\n };\n};" , "request" : {"url":"http://books.toscrape.com/"} }; fetch('https://chrome.agenty.com/functi...', { method: 'post', body: jsonBody, headers: { 'Content-Type': 'application/json' }, }) .then(res => res.json()) .then(json => console.log(json)); See the postman collection here - https://github.com/Agenty/agenty... or example scripts in sidebar.
Vikash Rathee
@chrismessina See this GIF using Postman -
πŸš€ Pierre-Henry πŸ’‘
Not too sure to understand exactly how easy it works. I will try to give it a try this afternoon. Anyway, like the idea πŸ˜‰
Vikash Rathee
@phenrysay It's scripting if you are a developer. But if not, no worries. We have point-and-click extension as well to setup scraping agents https://chrome.google.com/websto...