Introducing our new integration of Puppeteer with scraping agent to allow users to write custom puppeteer scripts in Node.js backed by Agenty cloud for performance and scaling.
Hi Product Hunters!
I will be live today for discussions, please ask question or share your feedback. I will try to respond all questions and plan the improvements in next version, if you have any feedback.
@chrismessina Just write a puppeteer script and use that to send POST request to https://chrome.agenty.com/function API to turn the website into API.
For example, here is an example code to navigate to site > extract 4 fields > return JSON
const fetch = require('node-fetch');
const jsonBody = {"code" : "// Read the `url` from request, goto the page, extract products\n// capture screenshot and return the results\n\nmodule.exports = async ({ page, request }) => {\n const response = await page.goto(request.url);\n console.log(response.status());\n const result = await page.evaluate(() => {\n const data = [];\n var products = document.querySelectorAll('.product_pod');\n for (var product of products) {\n data.push({\n product_name: product.querySelector('h3').textContent,\n product_price: product.querySelector('.price_color').textContent,\n product_availability: product.querySelector('.availability').textContent,\n product_image: \"http://books.toscrape.com\" + product.querySelector('.thumbnail').getAttribute(\"src\"),\n product_link: \"http://books.toscrape.com\" + product.querySelector('h3 > a').getAttribute(\"href\")\n });\n }\n return data;\n });\n\n await page.screenshot({ path: 'books-toscrape.png', fullPage: true });\n\n return {\n data: result,\n type: 'application/json'\n };\n};" , "request" : {"url":"http://books.toscrape.com/"} };
fetch('https://chrome.agenty.com/functi...', {
method: 'post',
body: jsonBody,
headers: { 'Content-Type': 'application/json' },
})
.then(res => res.json())
.then(json => console.log(json));
See the postman collection here - https://github.com/Agenty/agenty... or example scripts in sidebar.
idm.in