Web Scraping

MeetLogin is a powerful tool for ethical web scraping, allowing you to rotate browser identities and avoid bot detection.

Why Use MeetLogin for Scraping

Standard Playwright/Puppeteer setups are easily detected by bot protection services like Cloudflare, DataDome, and PerimeterX. MeetLogin's anti-detect engine passes these checks by presenting genuine browser fingerprints.

Basic Scraping Setup

import { MeetLogin } from 'meetlogin'
import { chromium } from 'playwright'

const client = new MeetLogin({ token: process.env.MEETLOGIN_API_KEY })

async function scrape(profileId, url) {
  // Launch the profile and get CDP URL
  const { wsUrl } = await client.profiles.start(profileId)

  // Connect Playwright
  const browser = await chromium.connectOverCDP(wsUrl)
  const context = browser.contexts()[0]
  const page = await context.newPage()

  try {
    await page.goto(url, { waitUntil: 'networkidle' })
    const data = await page.evaluate(() => {
      // Your extraction logic here
      return document.title
    })
    return data
  } finally {
    await browser.close()
    await client.profiles.stop(profileId)
  }
}

Rotating Profiles

For high-volume scraping, rotate across multiple profiles to distribute request load:

const profiles = await client.profiles.list({ workspaceId: WS_ID })
const urls = ['https://example.com/page1', '...', '...']

// Distribute URLs across profiles
await Promise.all(
  urls.map((url, i) => scrape(profiles[i % profiles.length].id, url))
)

Best Practices

• Add random delays between requests (1–5 seconds) to mimic human behavior.
• Use residential proxies for heavily protected sites.
• Respect robots.txt and sites' Terms of Service.
• Don't scrape more aggressively than a human could reasonably browse.