Usability Testing Report
Usability Testing Report
Usability Evaluation of Sky News Desktop Website: A Comparative Study with BBC News.
Usability Evaluation of Sky News Desktop Website: A Comparative Study with BBC News.
Go back
Go back
Project Type
Timeline
Tool
MsC coursework project
Qualtrics, Excel, Axure
Jan 2023 - April 2023 (4 months)
Introduction
During my MSc study, I conducted a usability evaluation of the Sky News website's desktop version. This project involved a complex design study with a mix of one-shot tests and A/B tests comparing Sky News with its competitor, BBC News. At the end of the project, I provided redesign suggestions based on my findings.
MsC coursework project
Project Type
MsC coursework project
Project Type
Jan 2022 - April 2023 (4 months)
Timeline
Jan 2022 - April 2023 (4 months)
Timeline
Qualtrics, Excel, Axure
Tool
Qualtrics, Excel, Axure
Tool
Qualtrics, Excel, Axure
Qualtrics, Excel, Axure
Tool
Jan 2022 - April 2023 (4 months)
Timeline
MsC coursework project
Project Type
Introduction
During my MSc study, I conducted a usability evaluation of the Sky News website's desktop version. This project involved a complex design study with a mix of one-shot tests and A/B tests comparing Sky News with its competitor, BBC News. At the end of the project, I provided redesign suggestions based on my findings.
01/ Background Research
Market research
Conducting usability testing with Sky News website compare with its competitor BBC News
According to a recent survey in 2022, Sky News is the third leading online news brand in the UK, with 13% of respondents using it weekly. However, a few usability issues could be solved from its desktop website. The problems lead me to conduct the usability report, with A/B testing with its biggest competitor, BBC News.
Market research
Conducting usability testing with Sky News website compare with its competitor BBC News
According to a recent survey in 2022, Sky News is the third leading online news brand in the UK, with 13% of respondents using it weekly. However, a few usability issues could be solved from its desktop website. The problems lead me to conduct the usability report, with A/B testing with its biggest competitor, BBC News.
01/ Background Research
02/ Heuristic evaluation
02/ Heuristic evaluation
02/ Heuristic evaluation
Heuristic evaluation
Heuristic evaluation
6 main usability issues found on the existing Sky News website desktop version
6 main usability issues found on the existing Sky News website desktop version
To frame the research questions for usability testing, I evaluated the Sky News website and identified six main issues to test further.
To frame the research questions for usability testing, I evaluated the Sky News website and identified six main issues to test further.
03/ Research Questions
03/ Research Questions
03/ Research Questions
Formulating research questions
Structuring the usability testing by turning the usability issues into research questions
The 6 main usability issues identified in the heuristic evaluation will be converted into 6 research questions to be tested during usability testing.
Formulating research questions
Structuring the usability testing by turning the usability issues into research questions
The 6 main usability issues identified in the heuristic evaluation will be converted into 6 research questions to be tested during usability testing.
03/ Research Questions
Formulating research questions
Structuring the usability testing by turning the usability issues into research questions
The 6 main usability issues identified in the heuristic evaluation will be converted into 6 research questions to be tested during usability testing.
04/ Usability Testing
04/ Usability Testing
Setting target group
Targeting participants aged 26-45 who read news regularly to participate in the test
I have targeted the participants to reduce the sample size for the usability test, which will be conducted in an A/B testing format. A survey was distributed to screen out participants who didn’t meet the requirements.
To avoid bias, participants will be divided into two groups and an A/B test will be conducted in different sequences.
Group 1: AAABBB
Group 2: BBBAAA
Creating task for participants
Design and evaluation of user tasks for Sky News and BBC News usability testing
Six tasks were designed for each participant using a complex method.
The first three tasks were one-shot tests on the Sky News website.
The other three tasks involved A/B testing and comparing Sky News and BBC News.
After each task, participants answered questions about their experience and preferences, followed by a short interview. Post-test surveys were provided to gather participants' overall experience.
Setting target group
Targeting participants aged 26-45 who read news regularly to participate in the test
I have targeted the participants to reduce the sample size for the usability test, which will be conducted in an A/B testing format. A survey was distributed to screen out participants who didn’t meet the requirements.
To avoid bias, participants will be divided into two groups and an A/B test will be conducted in different sequences.
Creating task for participants
Design and evaluation of user tasks for Sky News and BBC News usability testing
Six tasks were designed for each participant using a complex method.
The first three tasks were one-shot tests on the Sky News website.
The other three tasks involved A/B testing and comparing Sky News and BBC News.
After each task, participants answered questions about their experience and preferences, followed by a short interview. Post-test surveys were provided to gather participants' overall experience.
04/ Usability Testing
Setting target group
Targeting participants aged 26-45 who read news regularly to participate in the test
I have targeted the participants to reduce the sample size for the usability test, which will be conducted in an A/B testing format. A survey was distributed to screen out participants who didn’t meet the requirements.
To avoid bias, participants will be divided into two groups and an A/B test will be conducted in different sequences.
Creating task for participants
Design and evaluation of user tasks for Sky News and BBC News usability testing
Six tasks were designed for each participant using a complex method.
The first three tasks were one-shot tests on the Sky News website.
The other three tasks involved A/B testing and comparing Sky News and BBC News.
After each task, participants answered questions about their experience and preferences, followed by a short interview. Post-test surveys were provided to gather participants' overall experience.
05/ Result
05/ Result
05/ Result
Metrics to measure result
Apply metrics related to effectiveness, efficiency, and satisfaction to measure the usability test results
I have targeted the participants to reduce the sample size for the usability test, which will be conducted in an A/B testing format. A survey was distributed to screen out participants who didn’t meet the requirements.
To avoid bias, participants will be divided into two groups and an A/B test will be conducted in different sequences.
Effectiveness
Efficiency
Satisfication
Task success rate
Time on task
Assist count
Error count
System Usability Scale
Reaction card
Net Promoter Score
Overall satisfaction performance data
BBC News performances better than Sky News website based on the overall satisfaction data
As mentioned in the metrics above, satisfaction is measured by SUS, reaction cards, and NPS. After calculating all the means and numbers collected (as seen in the graph below), BBC News has higher customer satisfaction than Sky News.
System Usability Scale (SUS)
Net Promoter Score (NPS)
Sky News NPS score
Sky News NPS score
BBC News NPS score
BBC News NPS score
Overall effectiveness performance data
Participants failed to complete all the tasks on Sky News website
In tasks 2-4, a one-shot test on the Sky News website, participants failed to complete all tasks. The news categorization, topic organization, and navigation to the store locator need improvement.
Task completion rate on task 2-4.
Task completion rate on task 5-7.
Overall efficiency performance data
Participants took more time and needed more assistance to complete tasks on Sky News than on BBC News.
Efficiency, measured by time on task, assist count, and error rate, showed participants needed less help and made fewer errors on the BBC News website. Tasks 4 and 5 (finding the store locator and news on a specific date) had the most errors and are the top priorities for redesign.
Time-based efficiency task 2-4.
Time-based efficiency on task 5-7.
Error rate task 2-4.
Error rate on task 5-7.
Metrics to measure result
Apply metrics related to effectiveness, efficiency, and satisfaction to measure the usability test results
I have targeted the participants to reduce the sample size for the usability test, which will be conducted in an A/B testing format. A survey was distributed to screen out participants who didn’t meet the requirements.
To avoid bias, participants will be divided into two groups and an A/B test will be conducted in different sequences.
Overall effectiveness performance data
Participants failed to complete all the tasks on Sky News website
In tasks 2-4, a one-shot test on the Sky News website, participants failed to complete all tasks. The news categorization, topic organization, and navigation to the store locator need improvement.
Overall satisfaction performance data
BBC News performances better than Sky News website based on the overall satisfaction data
As mentioned in the metrics above, satisfaction is measured by SUS, reaction cards, and NPS. After calculating all the means and numbers collected (as seen in the graph below), BBC News has higher customer satisfaction than Sky News.
Overall efficiency performance data
Participants took more time and needed more assistance to complete tasks on Sky News than on BBC News.
Efficiency, measured by time on task, assist count, and error rate, showed participants needed less help and made fewer errors on the BBC News website. Tasks 4 and 5 (finding the store locator and news on a specific date) had the most errors and are the top priorities for redesign.
06/ Redesign suggestions
Redesign prioritization
Prioritizing Usability Issues for redesign focusing on categorization and clear information representation
Through usability testing, issues were identified and grouped into user interface features. After prioritizing the issues. High-severity issues will be targeted for redesign recommendations.
06/ Redesign suggestions
Redesign prioritization
Prioritizing Usability Issues for redesign focusing on categorization and clear information representation
Through usability testing, issues were identified and grouped into user interface features. After prioritizing the issues. High-severity issues will be targeted for redesign recommendations.
07/ Final Design
07/ Final Design
Redesigning Sky News: Applying Usability Testing Results
Based on the results from the usability testing and prioritizing the highest issues, I have applied new elements to redesign the Sky News website. A mockup wireframe has been developed using Axure.
Redesigning Sky News: Applying Usability Testing Results
Based on the results from the usability testing and prioritizing the highest issues, I have applied new elements to redesign the Sky News website. A mockup wireframe has been developed using Axure.
Project Takeaways
Project Takeaways
Project Takeaways
Usability testing can bring valuable insights
Highlighting the importance of user-centered design. Observing participant struggles and preferences allowed us to improve the user experience through informed decisions.
Usability testing can bring valuable insights
Highlighting the importance of user-centered design. Observing participant struggles and preferences allowed us to improve the user experience through informed decisions.
Prioritization of Issues
This way, we focused on the most important areas, making our redesign more effective and user-friendly.
Thank you for reading
Usability testing can bring valuable insights
Highlighting the importance of user-centered design. Observing participant struggles and preferences allowed us to improve the user experience through informed decisions.
Prioritization of Issues
This way, we focused on the most important areas, making our redesign more effective and user-friendly.
Thank you for reading
02/ Heuristic evaluation
Heuristic evaluation
6 main usability issues found on the existing Sky News website desktop version
To frame the research questions for usability testing, I evaluated the Sky News website and identified six main issues to test further.
05/ Result
BBC News NPS score
Task completion rate on task 2-4.
Task completion rate on task 5-7.
Overall efficiency performance data
Participants took more time and needed more assistance to complete tasks on Sky News than on BBC News.
Efficiency, measured by time on task, assist count, and error rate, showed participants needed less help and made fewer errors on the BBC News website. Tasks 4 and 5 (finding the store locator and news on a specific date) had the most errors and are the top priorities for redesign.
Metrics to measure result
Apply metrics related to effectiveness, efficiency, and satisfaction to measure the usability test results
I have targeted the participants to reduce the sample size for the usability test, which will be conducted in an A/B testing format. A survey was distributed to screen out participants who didn’t meet the requirements.
To avoid bias, participants will be divided into two groups and an A/B test will be conducted in different sequences.
Overall satisfaction performance data
BBC News performances better than Sky News website based on the overall satisfaction data
As mentioned in the metrics above, satisfaction is measured by SUS, reaction cards, and NPS. After calculating all the means and numbers collected (as seen in the graph below), BBC News has higher customer satisfaction than Sky News.
Overall effectiveness performance data
Participants failed to complete all the tasks on Sky News website
In tasks 2-4, a one-shot test on the Sky News website, participants failed to complete all tasks. The news categorization, topic organization, and navigation to the store locator need improvement.
Overall efficiency performance data
Participants took more time and needed more assistance to complete tasks on Sky News than on BBC News.
Efficiency, measured by time on task, assist count, and error rate, showed participants needed less help and made fewer errors on the BBC News website. Tasks 4 and 5 (finding the store locator and news on a specific date) had the most errors and are the top priorities for redesign.
Project Takeaways
Usability testing can bring valuable insights
Highlighting the importance of user-centered design. Observing participant struggles and preferences allowed us to improve the user experience through informed decisions.
Prioritization of Issues
This way, we focused on the most important areas, making our redesign more effective and user-friendly.
Thank you for reading
06/ Redesign suggestions
Redesign prioritization
Prioritizing Usability Issues for redesign focusing on categorization and clear information representation
Through usability testing, issues were identified and grouped into user interface features. After prioritizing the issues. High-severity issues will be targeted for redesign recommendations.
07/ Final Design
Redesigning Sky News: Applying Usability Testing Results
Based on the results from the usability testing and prioritizing the highest issues, I have applied new elements to redesign the Sky News website. A mockup wireframe has been developed using Axure.