Scrape Websites with Python, FastAPI & NoSQL
- Python 3.9 download - programming the logic.
- AstraDB sign up - highly performant and scalable database service by DataStax. AstraDB is a Cassandra NoSQL Database. Cassandra is used by Netflix, Discord, Apple, and many others to handle astonding amounts of data.
- Selenium docs - an automated web browsing experience that allows:
- Run all web-browser actions through code
- Loads JavaScript heavy websites
- Can perform standard user interaction like clicks, form submits, logins, etc.
- Requests HTML docs - we're going to use this to parse an HTML document extracted from Selenium
- Celery docs - Celery providers worker processes that will allow us to schedule when we need to scrape websites. We'll be using redis as our task queue.
- FastAPI docs - as a web application framework to Display and monitor web scraping results from anywhere
- Scraping How to scrape and parse data from nearly any website with Selenium & Requests HTML.
- Data models how to store and validate data with cassandra-driver, pydantic, and AstraDB.
- Worker & Scheduling how to schedule periodic tasks (ie scraping) integrated with Redis & AstraDB
- Presentation How to combine the above steps in as robust web application service
Reference Code
Setup your system.
Preflight checklist
- [] Install Selenium & Chromedriver - setup guide
- [] Install Redis - setup guide
- [] Create a virtual environment & install dependencies
- [] Setup an account with DataStax
- [] Create your first AstraDB and get API credentials
- [] Use cassandra-driver to verify your connection to AstraDB
Lessons
Welcome
0:00
Overview & Requirements
0:00
Project Setup
0:00
Start the Python & Cassandra Integration
0:00
Configure Python cassandra-driver
0:00
Your First Cassandra Model
0:00
Create Data using our Cassandra Model
0:00
Adding a New Column to existing Model
0:00
Using UUID1 as Primary Key
0:00
Using Jupyter with Cassandra Models
0:00
Using Pydantic for Data Validation and Cleaning
0:00
FastAPI & Environment Variables
0:00
FastAPI, Cassandra & Pydantic
0:00
Convert Cassandra UUID Field to Pydantic Datetime Str
0:00
Endpoint to Ingest Data for FastAPI & AstraDB
0:00
Celery, Redis & Basic Task Offload
0:00
Integrate Cassandra Driver with Celery
0:00
Running Periodic Tasks
0:00
Basic Scraping with Selenium
0:00
Using Javascript in Selenium to Perform an Endless Scroll
0:00
requests-html & Parsing Data
0:00
Implement the Scrape Client Parser
0:00
Putting it all together
0:00
Thank you
0:00