i coded a scraper !!!!!!!!!!
realvizierr's statistics
-----------------------------------
total upvotes : 895
total downvotes: -453
net votes : 442
-----------------------------------
most upvotes received: 36
"guess we'll never know"
-----------------------------------
most downvotes received: -54
"fun match, too bad the evil won"
-----------------------------------
here you can be the special person that gets one of the functions!
def parse(self, response):
# get total amount of pages of like posts a user has
page_links = response.css('a.btn.mod-page::attr(href)').getall()
# errm get last page lol
last_page_number = int(page_links[-1].split('=')[-1]) if page_links else 1
# go through all the pages
for page_number in range(1, last_page_number + 1):
url = f'/user/{self.username}/?page={page_number}'
yield response.follow(url, self.parse_user_page)