i wanted to edit my webscraper to basically get a user's most common saying, but obviously a person can say the same thing in multiple ways
how would i go about evaluating this? i was thinking of using either levenshtein or hamming distance since both have libraries supporting them
i obviously dont wanna slow down the scraping process too much when evaluating this, so how should i go about it?
trying to make a VLR fantasy league site, so converting the stats page - with relevant parameters - into csv file/mysql database. pretty low level but ive got it working alright now. using it for a stats page on my site, and to be able to have the data for when players want to have their own teams. makes it a lot more efficient than sending GET requests for everything client side.
def scrapeStats(url):
getResponse = requests.get(url)
#verifying successful request response
if getResponse.statuscode == 200:
#parse HTML content
doc = BeautifulSoup(getResponse.content, "html.parser")
#saving only table data
table = doc.find(class='wf-card mod-table mod-dark')
headers = []
rows = []
#extract and fill the header and row arrays with table data
for th in table.find_all('th'):
headers.append(th.text.strip())
for tr in table.find_all('tr')[1:]:
row = []
for td in tr.find_all('td'):
row.append(td.text.strip())
rows.append(row)
return headers, rows
else:
#in case of unsuccessful request response
print("Request Error")
return None, None
def clean_data(headers, rows):
#appending new team column
for row in rows:
player_name = row[0]
team_name = player_name.split()[-1]
row.append(team_name)
#deleting redundant agents column
agents_index = headers.index('Agents')
for row in rows:
del row[agents_index]
del headers[agents_index]
def save_to_csv(headers, rows, fileName):
with open(fileName, 'w', newline='') as csvFile:
writer = csv.writer(csvFile)
#write headers and rows
#create team header
writer.writerow(headers + ['Team'])
for row in rows:
writer.writerow(row)
print(f"Data has been saved to {fileName}")
def main():
#change this depending on the vlr stats page that you want it to scrape from
url = "https://www.vlr.gg/event/stats/1921/champions-tour-2024-masters-madrid?exclude=&min_rounds=0&agent=all"
#created CSV file name
fileName = 'vlr_stats.csv'
#scraping data
headers, rows = scrapeStats(url)
#check data is retrieved successfully
if headers and rows:
#clean data
clean_data(headers, rows)
#save data to CSV file
save_to_csv(headers, rows, fileName)
if name == "main":
main()
wdym in multiple ways? if you mean the saying is written between different types of texts then you just get the entire reply/comment.
If you're using python, look for the saying within the big block of text with something like:
(pretend the periods are spaces, I cant get this shit to indent)
Instances = []
for text_block in user_history:
....If string1 in text: #Returns true if there is any instance of the string you provided within the larger string (the block of text)
...........Instances.append(text):
No I meant for example the following strings:
"yay is good"
"y0y is good"
"good is y0y"
"good is yay"
Are practically the same to the human eye, but comparison wise they are different. Thats why I want to use levenshtein or hamming distance. But also the code you provide is kind of slow and I'm looking for a faster solution since I'll most likely be doing this for every users post
Haha yeah I apologize there's definitely more optimized ways to do it.
Would speed still be an issue if you ran the program concurrently using multiprocessing, with an instance for every user profile?
#definitely a way to automate and optimize this part but no time rn
look_for = ["yay is good", "y0y is good", "good is y0y", "good is yay", "good yay is", "good y0y is", "y0y good is", "yay good is"]
for comment in message_history:
...for instance in look_for:
......if instance in comment:
.........Do_thing()
It's an ugly solution but would this work for what you're trying to do?
Somewhat, it's a starting point but I think I'm still going to go with hamming distance or something because it requires less comparison I think.
I'm already doing the webscraping concurrently with around ~10-14 pages per second depending on my connection.
This honestly might get complicated too fast, might have to drop the idea ngl lol
Discord