This is the code that i have so far
for page in range(1, 5):
guitarPage
=requests.get('https://www.guitarguitar.co.uk/guitars/electric/page-'.format(page)).text
soup = BeautifulSoup(guitarPage, 'lxml')
# row = soup.find(class_='row products flex-row')
guitars = soup.find_all(class_='col-xs-6 col-sm-4 col-md-4 col-lg-3')
This is the actual loop to iterate over the products
for guitar in guitars:
title_text = guitar.h3.text.strip()
print('Guitar Name: ', title_text)
price = guitar.find(class_='price bold small').text.strip()
print('Guitar Price: ', price)
time.sleep(0.5)
The code so far only, runs through the same page, without moving on to the next page.
The structure of the URL of the website works around page-2,page-3 ++ and so on.
You have to add {} to your link. I have added also time module.
import requests
from bs4 import BeautifulSoup
import time
for page in range(1, 5):
guitarPage = requests.get('https://www.guitarguitar.co.uk/guitars/electric/page-{}'.format(page)).text
soup = BeautifulSoup(guitarPage, 'lxml')
# row = soup.find(class_='row products flex-row')
guitars = soup.find_all(class_='col-xs-6 col-sm-4 col-md-4 col-lg-3')
for guitar in guitars:
title_text = guitar.h3.text.strip()
price = guitar.find(class_='price bold small').text.strip()
print('Guitar Name: ', title_text, 'Guitar Price: ', price)
time.sleep(0.5)
Related
I am currently making a quiz project for school. The quiz is supposed to evaluate in the front-end, which works perfectly fine, and after that it should send the information to the backend. The backend should save the details to the database and after that it should send the information back to the front-end. The front-end should show a message with some details about how you performed compared to all the others, the information is gotten from the database.
So, the weird thing about all that is that it works sometimes and sometimes not. If I test it on localhost, the code works perfectly fine. The error only occurs on the Ionos server (I got a hosting contract so I do not have access to the console).
Here is the link to my website: https://tor-netzwerk-seminarfach2024.com/ . If you click on the upper left corner and then on the quiz button, you will get to the quiz. If I look at the console and network analytics, when the error occurs, the following message shows up:
Cross-Origin request blocked: The same source rule prohibits reading
the external resource on https://api. tor-netzwerk-seminarfach2024.
com/apache-test/. (Reason: CORS request failed).
https://api.tor-netzwerk-seminarfach2024.com/apache-test/ is my backend and https://tor-netzwerk-seminarfach2024.com/ is my front-end.
The settings file in Django is not the problem. To make it clear, the error does not occur always. Sometimes you have to go again to the website and try it one a few more times until you get the error.
If I look at the network analytics, then I see that the client request works as usual, but the answer field from the server is completely empty. Let's move on to the weirdest part of all: the Data actually gets saved in the Database, there is just sometimes no response?
It would be way to much for a minimum working product but here is the most important code. If you want to test it you can just visit the link to my Website:https://tor-netzwerk-seminarfach2024.com/
Server side with Django
from rest_framework.response import Response
from .models import Lead
import json
from .serializer import TestingSerializer
def updateDigits():
leadObj = Lead.objects.all()
currentScore = leadObj[0].sumScore; currentRequests = leadObj[0].sumRequests
valueBtnOne = leadObj[0].buttonOne; valueBtnTwo = leadObj[0].buttonTwo;
valueBtnThree = leadObj[0].buttonThree; valueBtnFour = leadObj[0].buttonFour;
valueBtnFive = leadObj[0].buttonFive; valueBtnSix = leadObj[0].buttonSix;
valueBtnSeven = leadObj[0].buttonSeven; valueBtnEight = leadObj[0].buttonEight;
valueBtnNine = leadObj[0].buttonNine;
return [valueBtnOne,valueBtnTwo,valueBtnThree,valueBtnFour,valueBtnFive,valueBtnSix,valueBtnSeven,valueBtnEight,valueBtnNine,currentScore,currentRequests]
#api_view(["POST"])
def home(request):
body_unicode = request.body.decode('utf-8')
body = json.loads(body_unicode)
result = body["Result"]
isItRight = body["whichOnesRight"]
ip_adress = get_client_ip(request)
queryset = Lead.objects.all()
if Lead.objects.all().exists():
currentValues = updateDigits()
queryset.update(
sumScore = result + currentValues[9],
sumRequests = 1 + currentValues[10],
buttonOne = currentValues[0] + isItRight[0],
buttonTwo = currentValues[1] + isItRight[1],
buttonThree = currentValues[2] + isItRight[2],
buttonFour = currentValues[3] + isItRight[3],
buttonFive = currentValues[4] + isItRight[4],
buttonSix = currentValues[5] + isItRight[5],
buttonSeven = currentValues[6] + isItRight[6],
buttonEight = currentValues[7] + isItRight[7],
buttonNine = currentValues[8] + isItRight[8],
)
currentValues = updateDigits()
else:
obj = Lead()
obj.save()
serializer = TestingSerializer(queryset[0], many =False)
return Response({**serializer.data, "myResult": result})
The Django model
from django.db import models
class Lead(models.Model):
sumScore = models.IntegerField(default=0)
sumRequests = models.IntegerField(default=0)
buttonOne = models.IntegerField(default=0)
buttonTwo = models.IntegerField(default=0)
buttonThree = models.IntegerField(default=0)
buttonFour = models.IntegerField(default=0)
buttonFive = models.IntegerField(default=0)
buttonSix = models.IntegerField(default=0)
buttonSeven = models.IntegerField(default=0)
buttonEight = models.IntegerField(default=0)
buttonNine = models.IntegerField(default=0)
serializer just serializes _ _ all _ _ ...
Front end with React
function evaluateAndCheckQuiz(){
let countNotChosen = howManyNotChosen()
if(countNotChosen){ answerQuestions(countNotChosen); return }
let points = evaluateQuiz()
document.getElementById("scoreText").style.display = "block"
document.getElementById("scoreNumber").textContent = points
return points
}
Calling sendAndGetQuizAverage with evaluateAndCheckQuiz as a parameter
function sendAndGetQuizAverage(points){
const myPostRequest = {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({"Result":points, "whichOnesRight": whichOnesRight}),
};
let data
fetch('https://api.tor-netzwerk-seminarfach2024.com/apache-test/', myPostRequest).then(
(response) => data = response.json()).then(
(data) => {showResultAnalyse(data)})
}
function showResultAnalyse(data){
console.log(data)
let averageScore = (data["sumScore"] / data["sumRequests"]).toFixed(2)
let myResult = data["myResult"]
changeLinksToStatistics([
data["buttonOne"],data["buttonTwo"],data["buttonThree"],
data["buttonFour"],data["buttonFive"],data["buttonSix"],
data["buttonSeven"],data["buttonEight"],data["buttonNine"]
],
data["sumRequests"]
)
swal(
{
text:`${checkWhichText(myResult,averageScore,data["sumRequests"] )}`,
icon: myResult > averageScore ?"success": "error",
button:"Oki doki",
}
);
}
please help me find the error as i didn’t understand for correctly :
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://www.imdb.com/chart/top/?ref_=nv_mv_250'
response = requests.get(url)
with open("imdb_top_250_movies.html", mode='wb') as file:
file.write(response.content)
soup = BeautifulSoup(response.content, 'lxml')
df_list = []
for movie in soup:
title = movie.find('td' , class_="titleColumn").find('a').contents[0]
year = movie.find('td' , class_="titleColumn").find('span').contents[0][1:-1]
user_rating = movie.find('td' , class_="ratingColumn imdbRating").find('strong').contents[0]
df_list.append({'title': title,
'year': int(year),
'user_ratings': float(user_rating)})
df = pd.DataFrame(df_list, columns = ['title', 'year', 'user_ratings'])
df
This is the error I got
TypeError Traceback (most recent call
last) Input In [125], in <cell line: 8>()
9 soup = BeautifulSoup(response.content, 'lxml')
10 df_list = []
---> 11 title = movie.find('td' , class_="titleColumn").find('a').contents[0]
12 year = soup.find('td' , class_="titleColumn").find('span').contents[0][1:-1]
13 user_rating = soup.find('td' , class_="ratingColumn imdbRating").find('strong').contents[0]
TypeError: find() takes no keyword arguments
Someone helped me with this answer as I wrote For incorrectly :
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://www.imdb.com/chart/top'
response = requests.get(url)
with open("imdb_top_250_movies.html", mode='wb') as file:
file.write(response.content)
soup = BeautifulSoup(response.content, 'lxml')
df_list = []
for movie in soup.find('tbody' , class_="lister-list").find_all('tr'):
Place = movie.find('td' , class_="titleColumn").contents[0][1:-len('.\n ')]
title = movie.find('td' , class_="titleColumn").find('a').contents[0]
year = movie.find('td' , class_="titleColumn").find('span').contents[0][1:-1]
user_rating = movie.find('td' , class_="ratingColumn imdbRating").find('strong').contents[0]
df_list.append({'place': Place,
'title': title,
'year': int(year),
'user_ratings': float(user_rating)})
df = pd.DataFrame(df_list, columns = ['place','title', 'year', 'user_ratings'])
df.style.hide(axis='index')
I am using discord_slash.utils.manage_components for help commands. I am trying to create help command in python.
https://cdn.discordapp.com/attachments/908053630425366538/940270195040198696/unknown.png
This my code
from discord_slash.utils.manage_components import create_select, create_select_option, create_actionrow
#cog_ext.cog_slash(name="help", description="Get the list of commands")
async def _help(self, ctx: SlashContext):
select = create_select(
options=[
create_select_option("General", value="General", emoji="🌐", default=True),
create_select_option("Fun", value="Fun", emoji="😄", default=False),
create_select_option("Admin", value="Admin", emoji="🛡️", default=False)
],
placeholder="Choose your option",
min_values=1, # the minimum number of options a user must select
max_values=1 # the maximum number of options a user can select
)
action_row = create_actionrow(select)
# Embed general
embed1 = discord.Embed(
title=
"<:mephisto:940249600969801748> **Mephisto General Commands**",
color=0xDC0000)
embed1.add_field(name="`/ping`", value="Return websocket ping")
embed1.add_field(name="`/invite`",
value="Invite bot to your server")
embed1.add_field(name="`/user-info [member]`",
value="Get a someone info")
embed1.add_field(
name="`/weather-info [city]`",
value="Tell us the weather of the city you want to know")
embed1.add_field(name="`/donut`",
value="Send ASCII art donut spinning")
embed1.add_field(name="`/server-info`", value="Server info")
embed1.add_field(name="`/info`",
value="Get information about Mephisto bot")
# Embed fun
embed2 = discord.Embed(
title=
"<:mephisto:940249600969801748> **Mephisto Fun Commands**",
color=0xDC0000)
embed2.add_field(name="`/joke`", value="Get a random joke")
embed2.add_field(name="`/meme`",
value="Get a random meme")
# Embed admin
embed3 = discord.Embed(
title=
"<:mephisto:940249600969801748> **Mephisto Admin Commands**",
color=0xDC0000)
embed3.add_field(name="`/clear [amount]`", value="Clear messages")
embed3.add_field(name="`/ban`",
value="Ban someone user")
embed3.add_field(name="`/Kick`",
value="Kick someone user")
pages = [embed1]
page0 = [embed1]
page1 = [embed2]
page2 = [embed3]
total_pages = [page0, page1, page2]
page = 0
category="General 🌐"
message = await ctx.send(embed=pages[page], components=[action_row])
while True:
try:
interaction = await self.bot.wait_for(
'interaction',
check=lambda inter: inter.message.id == message.id,
timeout=60
)
except asyncio.TimeoutError:
for row in action_row:
row.disable_components()
return await message.edit(content='Timed out!', components=action_row)
if isinstance(interaction.component, create_select):
action_row[0][0].default=False
create_select_option.default=True
pages=page1
page=0
try:
action_row[1][0].disabled=True
action_row[1][1].disabled=True
print("check")
except:
pass
await interaction.edit_origin(embed=pages[page], components = action_row)
Select option doesn't working
I couldn't understand what is the problem. Someone help me to solve this problem.
I want to be able to return a populated dash table based on the results from an input search. I've tried 2 methods so far - returning the entire DashTable in the callback output and returning the columns and data separately in the callback. Both options haven't been working for me. I've included the relevant code for each option and the error message that results from each:
Return the data and columns separately:
#app.callback(
[Output('table', 'data'),
Output('table', 'columns')],
[Input("button", "n_clicks")], state=[State('url', 'value')])
def update_table(n_click:int, url):
if n_click>1:
summary, table = summarizer(url)
columns=[{"name": i, "id": i, "deletable": True, "selectable": True} for i in table.columns]
table = table.to_dict('records')
return table, columns
else:
return [], []
The app.layout contains the following line
html.Div(dt.DataTable(id='table'))
The error message that results from this is:
Objects are not valid as a React child
The second approach was to pass in the entire DataTable through the callback and display it using just the html.Div in the layout like this
#app.callback(
Output('table', 'children'),
[Input("button", "n_clicks")], state=[State('url', 'value')])
def update_table(n_click:int, url):
if n_click>1:
summary, table = summarizer(url)
columns=[{"name": i, "id": i, "deletable": True, "selectable": True} for i in table.columns]
table = table.to_dict('records')
return dt.DataTable(data=table, columns=columns)
else:
return []
html.Div(id='table')
The corresponding error was
[Objects are not valid as a React child][2]
This error is confusing to me since it seems to be regarding the column definition however I can't pass in an array and the documentation asks for a dictionary.
Full code sample:
import dash
import dash_core_components as dcc
import dash_html_components as html
import dash_bootstrap_components as dbc
import dash_table as dt
from dash.dependencies import Input, Output, State
import sd_material_ui
from newspaper import Article
import gensim
from gensim.summarization import summarize
from dash.exceptions import PreventUpdate
from newspaper import fulltext
import requests
import pandas as pd
import yake
import nltk
from newsapi import NewsApiClient
leftSources = ["cnn", "buzzfeed", "the-washington-post", "bbc-news", "vice-news", "newsweek", "techcrunch", "reuters", "politico", "newsweek", "msnbc"]
rightSources = ["fox-news", "national-review", "new-york-magazine", "breitbart-news", "business-insider", "the-wall-street-journal", "bloomberg", "the-washington-times", "the-hill", "the-american-conservative"]
# importing CSS
external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css']
# similarArticleURL
getSimilarArticlesURL = "https://us-central1-secure-site-266302.cloudfunctions.net/getSimilarArticles?keywords="
getKeywordsURL = "https://us-central1-secure-site-266302.cloudfunctions.net/getKeyword?text="
getArticleTextURL = "https://us-central1-secure-site-266302.cloudfunctions.net/getArticleText?url="
allData = pd.DataFrame()
# instantiating dash application
app = dash.Dash(__name__, external_stylesheets=[dbc.themes.BOOTSTRAP])
server = app.server # the flask app
# helper functions
def generate_table(dataframe, max_rows=10):
return html.Table([
html.Thead(
html.Tr([html.Th(col) for col in dataframe.columns])
),
html.Tbody([
html.Tr([
html.Td(dataframe.iloc[i][col]) for col in dataframe.columns
]) for i in range(min(len(dataframe), max_rows))
])
])
app.layout = html.Div([
html.Div(html.H3("Brief.Me"), style={'font-weight':'bold','background-color':'darkorange', 'color':'white','text-align':'center'}),
html.Br(),
html.Br(),
dbc.Row([
dbc.Col(dbc.Input(id='url', type='url', size=30, placeholder="Type or copy/paste an URL"), width={'size':6, 'order':1, 'offset':3}),
dbc.Col(dbc.Button("Summarize", id='button', n_clicks=1, color="primary", className="mr-1"), width={'order':2})
]),
html.Br(),
# dbc.Row([
# dbc.Col(dcc.Loading(html.Div(html.Div(id="summary"), style={'font-weight':'bold'})), width={'size':6, 'offset':3})
# ]),
html.Div(id='table')
],
)
def fetch_similar_articles(keyword):
tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
newsapi = NewsApiClient(api_key='ce7482cbd40f4d90a8eea404e7702db6')
top_headlines = newsapi.get_top_headlines(q=keyword,
sources='bbc-news,the-wall-street-journal,the-washington-post,fox-news,bloomberg, vice-news, politico, reuters, the-hill',
language='en')
return top_headlines["articles"]
def fetch_article_text(url):
try:
article = Article(url)
article.download()
article.parse()
return article.text
except:
return None
def summarizer(url):
global allData
leftSummaries, rightSummaries = {}, {}
text = fetch_article_text(url)
main_summary = summarize(text)
keywords = extract_keywords(text)
urls = []
rightData, leftData, allData = get_articles_content(keywords)
rightDf, leftDf = pd.DataFrame(rightData), pd.DataFrame(leftData)
allSources = pd.concat([rightDf, leftDf], axis=1)
return main_summary, allData
def get_articles_content(keywords):
'''
This function will return a row of the dataframe where there is a title, source, url and summary.
'''
allResults, leftRows, rightRows = [], [], []
for keyword in keywords:
articleList = fetch_similar_articles(keyword)
for elem in articleList:
source = elem['source']
url = elem['url']
title = elem['title']
text = fetch_article_text(url)
if text is not None and len(text) > 1:
summary = summarize(text)
allResults.append({'title': title, 'url': url,'source': source, 'summary': summary})
if source in leftSources:
leftRows.append(pd.DataFrame({'title': title, 'url': url,'source': source, 'summary': summary}))
elif source in rightSources:
rightRows.append(pd.DataFrame({'title': title, 'url': url, 'source': source, 'summary': summary}))
allResults = pd.DataFrame(allResults)
return leftRows, rightRows, allResults
def extract_keywords_yake(text, phrase_length, num_keywords):
custom_kw_extractor = yake.KeywordExtractor(n=phrase_length, top=num_keywords)
keywords = custom_kw_extractor.extract_keywords(text)
return keywords
def extract_keywords(text):
'''
Returns a list of keywords given the article text.
'''
global getKeywordsURL
getKeywordsURL += text
keywordRes = extract_keywords_yake(text, 2, 5)
keywords = []
for pair in keywordRes:
keywords.append(pair[1])
return keywords
#app.callback( # Output('summary', 'children')
Output('table', 'children'),
[Input("button", "n_clicks")], state=[State('url', 'value')])
def update_table(n_click:int, url):
if n_click>1:
summary, table = summarizer(url)
columns=[{"name": i, "id": i, "deletable": True, "selectable": True} for i in table.columns]
table = table.to_dict('records')
return dt.DataTable(data=table, columns=columns)
else:
return [], []
if __name__ == '__main__':
app.run_server(debug=True, host='0.0.0.0', port=8080)
I a user list that pops up when a user types # into a text field. When this happens I want the user list to be filtered for every letter typed after the # symbol. So if they typed "#s" all usernames starting with an "s" would show. I'm recording the text typed after the # and assigning it as the text to filter by, but the filter won't act until two characters have been typed. Can someone tell me why this would be happening? Thank you for your help.
HTML/SLIM:
/ USER LIST
.user-list.list-group [ng-show="usersShow"]
a.user.list-group-item [ng-repeat="user in users | filter:formattedUser"]
strong
| {{user.username}}
| {{' : ' + (user | fullName)}}
/ COMMENT FORM
= form_for Comment.new, html: {action: nil, name: 'newCommentForm', 'ng-submit' => 'createComment($event, comment)'} do |f|
= f.text_field :text, placeholder: 'Add comment...',
'ng-model' => 'comment.text', 'ng-change' => 'showUserList(comment.text)'
span.input-group-btn
button.btn type="submit" Add Comment
JS:
$scope.showUserList = function(comment) {
var USERNAME_PATTERN = /(?:^|\s)#\S*$/;
var comment = comment || '';
var userMatch = comment.match(USERNAME_PATTERN) || '';
if(userMatch) {
$scope.userFormatted = userMatch[0].replace('#', '').trim();
}
$scope.usersShow = USERNAME_PATTERN.test(comment);
};