What you can realistically get

Before writing code, decide what you need:

• Search results: titles, prices, shop names, ratings, and whether a listing is an ad or a bestseller. This is the usual input for niche research.
• Listing details: description, variations, materials, favorites, shipping, and reviews.
• Shop data: all active listings for one shop, its sales count, and reviews. Good for competitor tracking.

Etsy offers three ways in: the official Open API v3, scraping the HTML, or a third-party API. Each one has real trade-offs.

Option 1: The official Etsy Open API v3

This is the cleanest way in, and you should try it first. You register an app at the Etsy developer portal and get a keystring (and a shared secret). Public endpoints such as active listing search only need the key in an x-api-key header. You don't need OAuth unless you're acting on behalf of a shop.

import os
import time
import requests

API = "https://openapi.etsy.com/v3/application"
# Recent Etsy docs ask for "keystring:shared_secret" here; older apps used the keystring alone.
HEADERS = {"x-api-key": os.environ["ETSY_API_KEY"]}

def search_active(keywords, max_items=300):
    """Yield active listings matching keywords, paginating with limit/offset."""
    offset, limit = 0, 100  # 100 is the max page size
    while offset < max_items:
        r = requests.get(
            f"{API}/listings/active",
            headers=HEADERS,
            params={"keywords": keywords, "limit": limit, "offset": offset,
                    "sort_on": "score"},
            timeout=30,
        )
        if r.status_code == 429:  # rate limited: back off and retry
            time.sleep(int(r.headers.get("retry-after", 5)))
            continue
        r.raise_for_status()
        results = r.json().get("results", [])
        if not results:
            break
        yield from results
        offset += limit
        time.sleep(0.2)  # stay well under the per-second limit

def price(listing):
    p = listing["price"]  # money object: amount / divisor
    return p["amount"] / p["divisor"], p["currency_code"]

for l in search_active("linen apron", max_items=200):
    amount, cur = price(l)
    print(l["listing_id"], f"{amount:.2f} {cur}", l["title"][:60])

Things that tripped me up:

1. Prices aren't floats. They come back as {amount, divisor, currency_code}. If you forget the divisor, a $24.00 apron shows up as 2400.
2. App approval takes a while. New keys can sit in a pending state, and the commercial-use review can take days. Personal-use keys come with low limits.
3. Rate limits. Etsy enforces a per-second limit and a daily quota on each key. Handle 429 responses with a backoff. Don't fire requests in a tight loop.
4. API ranking isn't site ranking. The order of listings/active results doesn't match what a shopper sees on etsy.com. Ads, bestseller badges, and personalized ranking are missing. If you're researching what ranks, this matters.
5. Shop sales and some review data are limited or missing for shops you don't own.

If the official API covers what you need, stop here. It's stable, it's permitted, and it won't break next month.

Option 2: Scraping the HTML

Most people end up here to get what shoppers actually see: ranked search pages, "Bestseller" and "Star Seller" badges, promoted listings, and "in 20 carts" demand signals.

The naive version looks like this:

import requests
from bs4 import BeautifulSoup

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/129.0 Safari/537.36")

def search_html(query, page=1):
    r = requests.get(
        "https://www.etsy.com/search",
        params={"q": query, "page": page},
        headers={"User-Agent": UA, "Accept-Language": "en-US,en;q=0.9"},
        timeout=30,
    )
    if r.status_code in (403, 429) or "captcha" in r.text.lower():
        raise RuntimeError(f"Blocked (HTTP {r.status_code})")
    soup = BeautifulSoup(r.text, "html.parser")
    out = []
    for card in soup.select("[data-listing-id]"):
        link = card.select_one("a[href*='/listing/']")
        title = card.select_one("h3")
        price = card.select_one(".currency-value")
        out.append({
            "id": card["data-listing-id"],
            "url": link["href"].split("?")[0] if link else None,
            "title": title.get_text(strip=True) if title else None,
            "price": price.get_text(strip=True) if price else None,
            "is_ad": "Ad" in card.get_text(" ", strip=True)[:200],
        })
    return out

Here's the honest part. From a datacenter IP this usually returns a 403 or a captcha page on the first request. Etsy sits behind serious bot protection that checks TLS fingerprints, headers, and IP reputation. Here's what makes things better or worse:

• IP type matters most. Residential IPs get much further than cloud ones. Even from a home connection, you'll get challenged after a few dozen fast requests.
• requests has a recognizable TLS fingerprint. Libraries like curl_cffi (with impersonate="chrome") help, but they're not a guarantee.
• Headless browsers (Playwright) get past more checks, but each page is slow and memory-heavy, and they still get flagged at volume.
• Selectors drift. Class names change. data-listing-id has held up longest for me, but plan on fixing your parser every so often.
• Locale and currency depend on your IP. Your US-dollar price tracker can quietly start logging euros if your proxy moves to Europe. Pin the currency, then check it on every row.
• Ads are mixed into organic results. If you're measuring rank, strip out the promoted cards first, or your position numbers are wrong.

Also read Etsy's terms and robots.txt before you scale this up. Scraping public pages at a polite rate for research is common, but hammering the site or republishing listing content is how people get into trouble.

Storing results so they're actually useful

Whatever source you use, store snapshots, not just the latest state. Price and rank changes over time are where the insight is:

import sqlite3, datetime

db = sqlite3.connect("etsy.db")
db.execute("""CREATE TABLE IF NOT EXISTS snapshots (
    listing_id TEXT, query TEXT, rank INTEGER, price REAL, currency TEXT,
    title TEXT, seen_at TEXT, PRIMARY KEY (listing_id, query, seen_at))""")

def save(query, rows):
    now = datetime.date.today().isoformat()
    db.executemany(
        "INSERT OR REPLACE INTO snapshots VALUES (?,?,?,?,?,?,?)",
        [(r["id"], query, i + 1, r["price"], r["currency"], r["title"], now)
         for i, r in enumerate(rows)],
    )
    db.commit()

Store one row per listing per query per day. With that, "which listings climbed for 'linen apron' this month" becomes a single SQL query.

Option 3: A scraping API

If you need ranked search pages with badges and ads, and you don't want to run proxies and fix selectors, a hosted API is the practical middle ground. I use ScrapeAtlas* for this. Its /v1/etsy/search endpoint returns one search page as JSON. That includes the ordered organic listing IDs, a card for each result (shop, current and original price, discount, rating, bestseller and ad flags), the promoted listings, and the total page count:

import os, requests

r = requests.get(
    "https://api.scrapeatlas.com/v1/etsy/search",
    headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
    params={"query": "linen apron", "page": 1,
            "sort": "most_relevant", "currency": "USD"},
    timeout=60,
)
data = r.json()
organic = [i for i in data["items"] if not i["isAd"]]
for rank, item in enumerate(organic, 1):
    print(rank, item["price"]["current"], item["seller"], item["title"][:50])
print("pages:", data["totalPages"], "next:", data["nextPage"])

The currency parameter fixes the locale problem described above. sort accepts price_low, price_high, highest_reviews, and most_recent. Related endpoints cover full listings (/v1/etsy/listing, with variations, shipping, favorites, and demand signals), shop listings, and reviews. Pagination works the same way: keep requesting nextPage until it's null. The rows also drop straight into the SQLite table above.

Which one should you pick?

• Official API: catalog-style data, long-running and permitted, and you can wait for key approval.
• DIY HTML: small, occasional jobs where you want full control and don't mind maintaining it.
• Hosted API: you need what shoppers see (rank, ads, badges) reliably and at volume.

Start with the official API, measure what's missing, and only add complexity when you need it.

*I build it.

In this guide we'll build a deal alert: a script that searches Marketplace near a location, collects new listings below a price you set, and remembers which ones it has already seen so you only get alerted about new ones.

Before you start: Only collect public listing data, keep your request rate low, and don't scrape personal data about sellers. Meta's terms restrict automated collection, and Meta has taken scrapers to court, so read the terms and know where you stand. Never automate a personal logged-in account; that's the quickest way to lose it.

Why requests + BeautifulSoup won't work here

If you curl a Marketplace search URL, you get a near-empty HTML shell. The listings are loaded by JavaScript from Facebook's internal /api/graphql/ endpoint, and the HTML class names are obfuscated (x1lliihq x6ikm8r ...) and change all the time.

That leaves two realistic DIY options:

1. Drive a real browser (Playwright) and read the rendered DOM
2. Drive a real browser and intercept the GraphQL responses it receives, which gives you clean JSON

Option 2 holds up much better, so that's the one we'll build.

Setup

pip install playwright
playwright install chromium

Step 1: Build the search URL

Marketplace search URLs look like this:

https://www.facebook.com/marketplace/{city_slug}/search?query=road%20bike&minPrice=100&maxPrice=600&daysSinceListed=1

• city_slug is a city name or numeric location ID that Facebook uses (e.g. nyc, london, sanfrancisco). Open Marketplace in your browser, pick a location, and copy it from the URL.
• minPrice, maxPrice filter by price
• daysSinceListed=1 restricts to fresh listings, which is exactly what you want for alerts

from urllib.parse import urlencode

def search_url(city, query, min_price=None, max_price=None, days=1):
    params = {"query": query, "daysSinceListed": days}
    if min_price is not None:
        params["minPrice"] = min_price
    if max_price is not None:
        params["maxPrice"] = max_price
    return f"https://www.facebook.com/marketplace/{city}/search?{urlencode(params)}"

Step 2: Intercept the GraphQL responses

Instead of parsing obfuscated HTML, listen for the JSON the page itself loads. Marketplace search results arrive in GraphQL responses that contain marketplace_search with feed_units.edges.

import json
from playwright.sync_api import sync_playwright

def walk(obj, key):
    """Yield every value stored under `key` anywhere in a nested JSON object."""
    if isinstance(obj, dict):
        for k, v in obj.items():
            if k == key:
                yield v
            yield from walk(v, key)
    elif isinstance(obj, list):
        for v in obj:
            yield from walk(v, key)

def extract_listings(payload):
    out = []
    for listing in walk(payload, "listing"):
        if not isinstance(listing, dict) or "id" not in listing:
            continue
        price = (listing.get("listing_price") or {}).get("amount")
        loc = (((listing.get("location") or {}).get("reverse_geocode")) or {})
        out.append({
            "id": listing["id"],
            "title": listing.get("marketplace_listing_title"),
            "price": float(price) if price else None,
            "city": loc.get("city"),
            "url": f"https://www.facebook.com/marketplace/item/{listing['id']}/",
        })
    return out

Using a recursive walk() instead of a hard-coded path like data.marketplace_search.feed_units.edges[i].node.listing is deliberate. Facebook reshuffles its GraphQL response shapes often, but the listing objects and field names have been fairly stable. Searching by key survives most reshuffles.

Step 3: Scroll to load more (that's the pagination)

Marketplace has no page numbers. It uses infinite scroll, and each scroll triggers another GraphQL request. So pagination means scrolling until you have enough results or nothing new arrives.

import random

def scrape(url, max_scrolls=8):
    found = {}

    def on_response(resp):
        if "/api/graphql" not in resp.url:
            return
        try:
            body = resp.text()
        except Exception:
            return
        # Responses are sometimes several JSON documents separated by newlines
        for line in body.splitlines():
            try:
                for item in extract_listings(json.loads(line)):
                    found[item["id"]] = item
            except json.JSONDecodeError:
                continue

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            locale="en-US",
            viewport={"width": 1366, "height": 900},
        )
        page.on("response", on_response)
        page.goto(url, wait_until="domcontentloaded")
        page.wait_for_timeout(4000)

        # Close the "Log in" dialog if it appears
        close = page.locator('div[aria-label="Close"]')
        if close.count():
            close.first.click()

        last = 0
        for _ in range(max_scrolls):
            page.mouse.wheel(0, random.randint(2500, 4000))
            page.wait_for_timeout(random.randint(2000, 4000))
            if len(found) == last:
                break  # nothing new loaded: end of results or wall
            last = len(found)

        browser.close()
    return list(found.values())

Note that listings in the initial page load are often embedded in <script> tags rather than fetched via GraphQL. If your first batch is missing, also run extract_listings() on JSON blobs pulled from page.content().

Step 4: Dedupe with SQLite and alert on new deals

import sqlite3

def new_listings(listings, db="marketplace.db"):
    con = sqlite3.connect(db)
    con.execute("""CREATE TABLE IF NOT EXISTS seen (
        id TEXT PRIMARY KEY, title TEXT, price REAL, city TEXT, url TEXT,
        first_seen TEXT DEFAULT CURRENT_TIMESTAMP)""")
    fresh = []
    for l in listings:
        cur = con.execute(
            "INSERT OR IGNORE INTO seen (id, title, price, city, url) VALUES (?,?,?,?,?)",
            (l["id"], l["title"], l["price"], l["city"], l["url"]),
        )
        if cur.rowcount:
            fresh.append(l)
    con.commit()
    con.close()
    return fresh

if __name__ == "__main__":
    url = search_url("sanfrancisco", "road bike", min_price=150, max_price=600)
    for l in new_listings(scrape(url)):
        print(f"NEW: ${l['price']:.0f}  {l['title']}  {l['url']}")

Run it every 30–60 minutes with cron, and swap the print for a Telegram bot, a Discord webhook, or an email.

The pitfalls (honestly)

• The login wall. Logged-out sessions get a modal and fewer results after a few scrolls. Closing the modal works for a while; eventually you'll see fewer results or a forced login. Don't fix this with your personal account.
• Datacenter IPs. From a cloud VPS, Facebook often serves a login redirect immediately. Residential IPs work much better.
• Headless detection. Default headless Chromium is easy to fingerprint. Realistic viewport and locale settings help; going further means stealth plugins and an endless cat-and-mouse game.
• Location drift. Results depend on the location tied to your session/IP, not just the city in the URL. Check the city field on what you collect.
• Shape changes. Log a warning when a run returns zero listings. That's almost always a response-shape change, not a lack of bikes.

Skipping the browser entirely

If keeping a headless browser alive is more than you want to maintain, ScrapeAtlas* has a Facebook Marketplace search endpoint that takes the query, coordinates and the same filters (price range, radius, condition, date listed) and returns JSON with a cursor for the next page:

import os
import requests

def search_api(query, lat, lng, max_price=None, pages=3):
    listings, cursor = [], None
    for _ in range(pages):
        params = {"query": query, "lat": lat, "lng": lng,
                  "radius_km": 40, "date_listed": "last_24_hours"}
        if max_price:
            params["max_price"] = max_price
        if cursor:
            params["cursor"] = cursor
        data = requests.get(
            "https://api.scrapeatlas.com/v1/facebook/marketplace/search",
            headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
            params=params,
            timeout=60,
        ).json()
        if data.get("status") in ("failed", "challenged"):
            raise RuntimeError(data.get("requestId"))
        listings.extend(data.get("listings", []))
        if not data.get("has_next_page"):
            break
        cursor = data.get("cursor")
    return listings

print(len(search_api("road bike", 37.7749, -122.4194, max_price=600)))

You'd map those listings into the same new_listings() dedupe step, and the rest of the alert pipeline doesn't change.

Wrap-up

• Marketplace is JS-rendered, so use a browser and intercept GraphQL JSON rather than parsing HTML.
• Pagination = scrolling; stop when nothing new arrives.
• Dedupe by listing ID in SQLite so alerts only fire once.
• Expect the login wall and IP blocks to be your real maintenance cost.

What are you hunting for on Marketplace? Tell me in the comments. I'm curious which categories people actually automate.

*I build it.