Project 84 - Testing YouTube Shorts
The number 84 does not have any connection to Orwell's vision of the future. Instead, it signifies the total count of YouTube shorts created for this project. These 84 distinct YouTube shorts each convey a fact about a species. Seven species are featured, with each species having 12 facts. The shorts have been released over a 7-day period, spanning from 9 AM to 8 PM.
The Plan
To gain new subscribers, YouTube Shorts has proven to be the most efficient approach. However, it's of fundamental importance that a published Short garners enough views shortly after publication to generate interest and draw in new subscribers.
In the past, we have noticed a significant variability in the performance of published Shorts, both in terms of views within the first 24 hours and total views since publication. Our data collection spans an extended timeframe, and not all hours are adequately represented. Our objective is to gain a more comprehensive understanding of the impact of the timing of Short publication by releasing a substantial number of YouTube Shorts within a condensed timeframe.
We've been publishing Shorts that feature the captivating behaviors of specific species, incorporating the authentic sound recorded during video capture, or background music. Some of these Shorts have performed quite well. However, scaling up production of this type of Shorts has proven challenging.
Our new Shorts format will experiment with showcasing original clips of specific species, accompanied by intriguing facts. These facts will be presented as initial statements, which can be either true or false, followed by concise explanations. The primary aim of this innovative format is not only to spotlight the unique behaviors of these species but also to enhance the viewer's experience through a voice-over commentary.
Workflow Production Shorts
List of Species
The species chosen for this project are those that have piqued the interest of our audience and for which we have a wealth of video clips and photographs:
- Common Snapping Turtle (Chelydra serpentina)
- Eastern Grey Squirrel (Sciurus carolinensis)
- Great Blue Heron (Ardea herodias)
- Green Frog (Lithobates clamitans)
- Red-winged Blackbird (Agelaius phoeniceus)
- Humpback Whale (Megaptera novaeangliae)
- Wood Duck (Aix sponsa)
Generating Collection of Facts
We are utilizing current Edge Chat and ChatGPT 3.5 feature to generate 12 facts about a specific species, following these instructions:
I want to create videos presenting facts about name of species but I am looking into a special format. It starts with a question which asks if a certain fact is true or false about name of species and then elaborate in 2 or 3 sentences about the fact. Only provide questions which can answered with yes or no. Make sure it is a question with question mark and not a statement. When the item contains measurements use metric units. Return 10 different items ordered from simple fact to more complex.
After receiving the initial 10 facts, we request an additional 10. Once we have gathered 20 facts, we review them and exclude any that are incorrect, questionable, or otherwise unusable. If necessary, we request another set of 10 facts. The final collection of 12 facts is then reviewed, corrected, and revised to conform to the ultimate format. Finally, we review the orthography and style using ChatGPT 3.5, following these instructions:
Correct orthography and style: fact question and explanation
The final 12 facts are incorporated into the species document in Obsidian MD and published in our website Fauna Flora Photography
Voice Over Recording
All facts, species by species, are recorded in a single session in DaVinci Resolve Fairlight using an SM7B microphone connected to a Focusrite 2i2 via a CL-1. The recordings for each species are then segmented into individual facts, with a distinct color applied to each segment.
Next, each fact is further segmented, with a cut made after the question and between each sentence of the explanation. This process enables us to trim the recordings, eliminate pauses during the recording, and remove sentences that have been recorded multiple times.
Full Length Video and Shorts Timelines
Two timelines are created for each species.
First, we create a timeline with the standard horizontal video format. In this timeline, we include header text, a number associated with each fact, text for the question, and the yes or no signal. Each fact is accompanied by video recordings of the species or photos. The video recordings are enhanced using various techniques such as stabilization, zoom, speed adjustments, and color correction. Fairlight is employed to balance the audio from the video recordings, background music, and voice-over. This timeline will be exported as a full-length video and published.
The second timeline is created in the vertical format suitable for shorts, and all elements from the full-length video timelines are duplicated. However, the sound mix won't carry over, so it needs to be re-adjusted, at least for the initial shorts timeline. Subsequent shorts timelines for other species are generated by duplicating a previous shorts timeline that includes sound mix adjustments. The videos are meticulously reviewed to ensure that the most captivating parts of the video are visible in the narrower vertical format. By employing "mark in" and "mark out" points, the 12 shorts per species are exported.
Upload to YouTube
Before commencing the upload, we developed a short Python script to randomly assign a schedule to each short. The 84 shorts are released over the course of 7 days, with one short published every hour from 9 am to 8 pm. This step is crucial to eliminate any bias based on personal favorites. All text, including questions and explanations, is stored in a spreadsheet and is now linked with the corresponding schedule, tags, and hashtags. Uploading the YouTube shorts is an efficient process since all the data for each short has been prepared in advance.
The full-length videos for each species were published from Sunday, September 24, 2023, to Saturday, September 30, 2023, with one video released per day at a randomly selected hour between 10 AM and 3 PM. The shorts, on the other hand, were published from Monday, October 1, 2023, to Saturday, October 8, 2023, with 12 shorts released each day between 9 AM and 8 PM.
Data Collection and Analytics
YouTube Studio is used to monitor schedule of the shorts and to collect number of views during the first 24hrs.
We collect data points, including views, likes, and dislikes, by using a Python script that utilizes YouTube API v3. While we retrieve data for all videos, shorts, and live streams in our YouTube channel, our analysis is limited to the shorts that are part of this experiment.
The YouTube API v3 does not offer queries that allow for data retrieval within a specific time frame. As a result, shorts published near the end of the test period have fewer days to accumulate views compared to videos published near the start of the test period. However, unless a particular short goes viral, and data retrieval occurs at least four weeks after the last short was published, this issue can be safely disregarded. We identify viral videos using the modified Z-score method and remove them from the dataset.
We categorize views into three distinct collections: one based on the day of the week of publication, one based on the hour of publication, and one based on species. To assess whether the datasets within each collection significantly differ from one another, we employ two statistical tests for non-normalized distributions: the Kruskal-Wallis test and the Mann-Whitney test. The Kruskal-Wallis test helps us determine whether there are statistically significant differences among the datasets, while the Mann-Whitney test provides insights into which specific datasets, if any, exhibit significant differences (both tests use a significance level of 0.05). Additionally, we visualize the data through plots for each collection.
Observations and Notations
It is of high importance that, during an experiment, there is sufficient time and resources available to observe the participants in an experiment, and to record any extraordinary events. These observations are invaluable as they may reveal limitations of the experiment or inspire new ideas, hypotheses, and experiments. (Prof. Hans Kummer)
- The one short with an X in the thumbnail gets almost 0 initial views. Due to the thumbnail?
- There is an immediate peak of views, starting 15 minutes after publishing a short, and another, larger peak, 6 hours after publishing a short. Shorts published after 4pm behave differently though.
- Shorts with Humpback Whales don't get initial exposure.
- Miss-scheduled the Humpback Whale short Tuesday 5 pm and corrected by publishing it manually 17:10. Is this the reason that it converts to the most viewed short?
- It may be that publishing a new short about a species triggers views of a previously published short about the same species?
- Publishing shorts messes up views on videos
- There may be an impact of performance of a short by the performance of the short published an hour ago.
- Turns out that Google YouTube API isn't providing data of all 84 shorts. Fixed.
- In retrospect, looking at the colored deployment schedule, I notice a bias.
Results
The statistical analysis has been done with the count of views during the first 24hrs of each short using the browser version of YouTube studio analytics.
The choice to employ the Kruskal-Wallis test over the ANOVA one-way test was informed by the outcomes of the Shapiro test, which were used to evaluate the normality of data within each category and group. The results of the Shapiro test indicated that several datasets across different groups did not follow a normal distribution, which consequently led to the preference for the Kruskal-Wallis test.
Kruskal-Wallis test results for datasets by weekdays, hours and species:
- There is no statistically significant difference between weekdays. Kruskal-Wallis chi^2: 6.12; p-value: 0.408960; fd: 6
- There is statistically significant difference between hours. Kruskal-Wallis chi^2: 41.68; p-value: 0.000018; fd: 11
- There is statistically significant difference between species. Kruskal-Wallis chi^2: 13.85; p-value: 0.031358; fd: 6
Although, there is no statistical significant difference for weekdays, Tuesday, Thursday and Friday are performing better than other days. Better performing hours of a day are noon, 4 PM and 5 PM. Best performing species is the Humpback Whale (Megaptera novaeangliae).
Post-hoc Dunn Test for datasets by hours:
| 09:00 | 10:00 | 11:00 | 12:00 | 13:00 | 14:00 | 15:00 | 16:00 | 17:00 | 18:00 | 19:00 | 20:00 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 09:00 | - | 0.5874 | 0.8953 | 0.1170 | 0.9171 | 0.9607 | 0.9694 | 0.1740 | 0.6650 | 0.0057 | 0.0040 | 0.4173 |
| 10:00 | 0.5874 | - | 0.5002 | 0.3054 | 0.6610 | 0.6218 | 0.5612 | 0.4141 | 0.9127 | 0.0009 | 0.0006 | 0.1758 |
| 11:00 | 0.8953 | 0.5002 | - | 0.0893 | 0.8137 | 0.8565 | 0.9258 | 0.1360 | 0.5724 | 0.0085 | 0.0060 | 0.4967 |
| 12:00 | 0.1170 | 0.3054 | 0.0893 | - | 0.1433 | 0.1289 | 0.1083 | 0.8350 | 0.2566 | 0.0000 | 0.0000 | 0.0174 |
| 13:00 | 0.9171 | 0.6610 | 0.8137 | 0.1433 | - | 0.9563 | 0.8867 | 0.2094 | 0.7423 | 0.0041 | 0.0029 | 0.3600 |
| 14:00 | 0.9607 | 0.6218 | 0.8565 | 0.1289 | 0.9563 | - | 0.9301 | 0.1902 | 0.7012 | 0.0049 | 0.0034 | 0.3895 |
| 15:00 | 0.9694 | 0.5612 | 0.9258 | 0.1083 | 0.8867 | 0.9301 | - | 0.1622 | 0.6374 | 0.0064 | 0.0045 | 0.4396 |
| 16:00 | 0.1740 | 0.4141 | 0.1360 | 0.8350 | 0.2094 | 0.1902 | 0.1622 | - | 0.3543 | 0.0000 | 0.0000 | 0.0300 |
| 17:00 | 0.6650 | 0.9127 | 0.5724 | 0.2566 | 0.7423 | 0.7012 | 0.6374 | 0.3543 | - | 0.0014 | 0.0009 | 0.2134 |
| 18:00 | 0.0057 | 0.0009 | 0.0085 | 0.0000 | 0.0041 | 0.0049 | 0.0064 | 0.0000 | 0.0014 | - | 0.9084 | 0.0510 |
| 19:00 | 0.0040 | 0.0006 | 0.0060 | 0.0000 | 0.0029 | 0.0034 | 0.0045 | 0.0000 | 0.0009 | 0.9084 | - | 0.0388 |
| 20:00 | 0.4173 | 0.1758 | 0.4967 | 0.0174 | 0.3600 | 0.3895 | 0.4396 | 0.0300 | 0.2134 | 0.0510 | 0.0388 | - |
In bold are p-values less than 0.05 rejecting the hypothesis that the compared datasets are equally distributed.
Post-hoc Dunn Test for datasets by species:
| Heron | Turtle | Squirrel | Frog | Blackbird | Whale | Duck | |
|---|---|---|---|---|---|---|---|
| Heron | - | 0.1750 | 0.0191 | 0.0402 | 0.0005 | 0.7157 | 0.0974 |
| Turtle | 0.1750 | - | 0.3232 | 0.4871 | 0.0318 | 0.0853 | 0.7631 |
| Squirrel | 0.0191 | 0.3232 | - | 0.7695 | 0.2462 | 0.0068 | 0.4924 |
| Frog | 0.0402 | 0.4871 | 0.7695 | - | 0.1463 | 0.0157 | 0.6939 |
| Blackbird | 0.0005 | 0.0318 | 0.2462 | 0.1463 | - | 0.0001 | 0.0649 |
| Whale | 0.7157 | 0.0853 | 0.0068 | 0.0157 | 0.0001 | - | 0.0432 |
| Duck | 0.0974 | 0.7631 | 0.4924 | 0.6939 | 0.0649 | 0.0432 | - |
In bold are p-values less than 0.05 rejecting the hypothesis that the compared datasets are equally distributed.
Additional data points around Short format performance:
- Subscriptions/ 1k Views: 1.38 (vs 2.27)
- Geography: CA 41.7%, USA 21.3%, Others 37.0% (vs CA 0.9%, USA 6.4%, Others 92.3%)
Conclusions
Before drawing conclusions and rushing into action, let's take a closer look at the dynamics of YouTube, particularly within the realm of YouTube Shorts and the niche of wildlife videos and shorts. YouTube is a highly dynamic ecosystem with numerous variables that change almost daily: from other users publishing videos to YouTube engineers modifying the algorithms that determine which clips are shown to audiences. Patterns you observed last week may not hold true next week. A video that may perform poorly today could perform exceptionally well tomorrow. Let's keep this in mind as we examine the results of this experiment. Additionally, we must consider that each topic niche and time zone may exhibit unique behaviors.
Based on the results of this study, I plan to publish YouTube Shorts in the near future, primarily on Thursdays and Fridays but never after 5 PM. Ideally, I'll aim for either noon or 4 PM. Unfortunately, the cost of filming Humpback Whale (Megaptera novaeangliae) is significantly higher than for any of the other species we tested. The chances of encountering a Common Snapping Turtle (Chelydra serpentina) are quite low. Regrettably, common species like Eastern Grey Squirrel (Sciurus carolinensis) and Red-winged Blackbird (Agelaius phoeniceus) aren't as popular. Sill, predicting the popularity of a species with the collected data remains challenging. Therefore, despite being aware of certain audience preferences, I will continue producing shorts featuring any species whenever I'm able to capture interesting behavior.
Regarding the performance of the new format, we must consider that we are comparing apples to oranges for different reasons. Nevertheless, we can conclude that, although a key metric, subscriptions/views, is lower, the geographical reach is encompassing our primary markets, which we attribute to the English voiceover. Thus, the acquired subscriptions include a proportionally higher number of the audience from the primary market we are interested in. However, we consider subscriptions generated by published shorts to be of less value since many of them may have been generated accidentally by users clicking the subscribe button.
Although the assignment of deployment slots has been done randomly, I have observed certain patterns that may impact the validity of the results. Specifically, the majority of Red-winged Blackbird (Agelaius phoeniceus) shorts (9 out of 12) were published during the last third of the day. Given that this species is less popular, this uneven distribution could have influenced the results. Additionally, there are days when a particular species is not represented, such as Wednesdays and Sundays, which have no Great Blue Heron (Ardea herodias) shorts. If I were to repeat this experiment, I would employ a different approach to allot deployment slots, ensuring that there is at least one short for each species every hour, and not exceeding two shorts of the same species in a single day.
Another limitation to take into consideration is the issue with the deployment of the Humpback Whale (Megaptera novaeangliae) short on Tuesday at 5 PM. This video was the only one that was scheduled erroneously, and I had to switch to manual publication, which occurred 10 minutes after the originally scheduled time. It has garnered the highest number of views among all 84 videos, potentially skewing the data in favor of Tuesday as a preferred day and 5 PM as the ideal hour for publishing shorts. I cannot rule out the possibility that its performance was enhanced by the manual publication.
Would I conduct this experiment again? Yes, and I am planning to repeat it at least once per year. Publishing these shorts has not only generated subscribers in the preferred market but has also provided valuable insights into the performance of shorts published at various hours and on different days.
Appendix: Deployment Schedules
Schedule Videos
Schedule start Sunday Sep 24.
| Sep 24 | Sep 25 | Sep 26 | Sep 27 | Sep 28 | Sep 29 | Sep 30 | |
|---|---|---|---|---|---|---|---|
| 10 AM | Great Blue Heron | Snapping Turtle | |||||
| 11 AM | Humpback Whale | ||||||
| 12 AM | |||||||
| 1 PM | Grey Squirrel | Green Frog | |||||
| 2 PM | Wood Duck | ||||||
| 3 PM | Red-winged Blackbird |
Among all videos published in 7 days. Bold: Leader. Italic: Tail-end.
Schedule Shorts
Schedule start Monday Oct 2, 2023.
| Oct 2 | Oct 3 | Oct 4 | Oct 5 | Oct 6 | Oct 7 | Oct 8 | |
|---|---|---|---|---|---|---|---|
| 9 AM | Great Blue Heron 7 | Great Blue Heron 8 | Wood Duck 10 | Wood Duck 8 | Wood Duck 7 | Humpback Whale 12 | Green Frog 11 |
| 10 AM | Great Blue Heron 11 | Common Snapping Turtle 1 | Humpback Whale 5 | Great Blue Heron 10 | Eastern Grey Squirrel 6 | Green Frog 4 | Humpback Whale 11 |
| 11 AM | Common Snapping Turtle 12 | Humpback Whale 10 | Red-winged Blackbird 4 | Great Blue Heron 12 | Humpback Whale 7 | Common Snapping Turtle 7 | Humpback Whale 3 |
| 12 AM | Eastern Grey Squirrel 12 | Humpback Whale 9 | Wood Duck 9 | Great Blue Heron 2 | Green Frog 6 | Great Blue Heron 9 | Common Snapping Turtle 5 |
| 1 PM | Eastern Grey Squirrel 1 | Common Snapping Turtle 2 | Wood Duck 4 | Common Snapping Turtle 6 | Wood Duck 5 | Common Snapping Turtle 10 | Eastern Grey Squirrel 10 |
| 2 PM | Green Frog 8 | Wood Duck 6 | Red-winged Blackbird 3 | Great Blue Heron 3 | Green Frog 9 | Humpback Whale 2 | Wood Duck 3 |
| 3 PM | Eastern Grey Squirrel 7 | Common Snapping Turtle 4 | Eastern Grey Squirrel 9 | Eastern Grey Squirrel 5 | Eastern Grey Squirrel 2 | Eastern Grey Squirrel 4 | Common Snapping Turtle 8 |
| 4 PM | Great Blue Heron 5 | Green Frog 7 | Green Frog 10 | Common Snapping Turtle 3 | Humpback Whale 1 | Red-winged Blackbird 1 | Wood Duck 1 |
| 5 PM | Red-winged Blackbird 12 | Humpback Whale 8 | Humpback Whale 4 | Wood Duck 11 | Green Frog 2 | Humpback Whale 6 | Green Frog 12 |
| 6 PM | Red-winged Blackbird 11 | Red-winged Blackbird 10 | Eastern Grey Squirrel 8 | Wood Duck 2 | Red-winged Blackbird 2 | Great Blue Heron 1 | Red-winged Blackbird 7 |
| 7 PM | Eastern Grey Squirrel 3 | Wood Duck 12 | Red-winged Blackbird 5 | Green Frog 5 | Great Blue Heron 6 | Red-winged Blackbird 8 | Common Snapping Turtle 11 |
| 8 PM | Great Blue Heron 4 | Red-winged Blackbird 9 | Eastern Grey Squirrel 11 | Common Snapping Turtle 9 | Red-winged Blackbird 6 | Green Frog 1 | Green Frog 3 |
Among the shorts published in a day. Bold: Leader. Italic: Tail-end.
Appendix: Data Views First 24hrs
Views First 24hrs Normalized
| MON | TUE | WED | THU | FRI | SAT | SUN | Total | |
|---|---|---|---|---|---|---|---|---|
| 09:00 | 0.0126 | 0.0054 | 0.0018 | 0.0039 | 0.0030 | 0.0051 | 0.0040 | 0.0357 |
| 10:00 | 0.0183 | 0.0087 | 0.0095 | 0.0099 | 0.0000 | 0.0020 | 0.0444 | 0.0927 |
| 11:00 | 0.0014 | 0.0049 | 0.0019 | 0.0114 | 0.0465 | 0.0011 | 0.0027 | 0.0699 |
| 12:00 | 0.0150 | 0.0138 | 0.0179 | 0.0187 | 0.0583 | 0.0273 | 0.0008 | 0.1518 |
| 13:00 | 0.0062 | 0.0033 | 0.0121 | 0.0168 | 0.0012 | 0.0026 | 0.0028 | 0.0451 |
| 14:00 | 0.0010 | 0.0047 | 0.0043 | 0.0555 | 0.0012 | 0.0029 | 0.0139 | 0.0834 |
| 15:00 | 0.0198 | 0.0159 | 0.0005 | 0.0030 | 0.0011 | 0.0051 | 0.0032 | 0.0487 |
| 16:00 | 0.0154 | 0.0145 | 0.0060 | 0.0950 | 0.1118 | 0.0054 | 0.0027 | 0.2508 |
| 17:00 | 0.0012 | 0.1608 | 0.0077 | 0.0059 | 0.0011 | 0.0104 | 0.0076 | 0.1946 |
| 18:00 | 0.0008 | 0.0001 | 0.0001 | 0.0000 | 0.0000 | 0.0001 | 0.0000 | 0.0011 |
| 19:00 | 0.0001 | 0.0002 | 0.0000 | 0.0001 | 0.0000 | 0.0000 | 0.0000 | 0.0004 |
| 20:00 | 0.0095 | 0.0017 | 0.0008 | 0.0079 | 0.0046 | 0.0007 | 0.0007 | 0.0259 |
| Total | 0.1014 | 0.2339 | 0.0624 | 0.2280 | 0.2289 | 0.0626 | 0.0828 | 1.0000 |
Values have been normalized by dividing count of views of a short with sum of all views. Total views during the first 24hrs of each short is 15'321.
Views First 24hrs Percentage
| Species | % Views |
|---|---|
| Humpback Whale (Megaptera novaeangliae) | 42.05% |
| Great Blue Heron (Ardea herodias) | 18.41% |
| Common Snapping Turtle (Chelydra serpentina) | 15.66% |
| Green Frog (Lithobates clamitans) | 9.71% |
| Wood Duck (Aix sponsa) | 6.71% |
| Eastern Grey Squirrel (Sciurus carolinensis) | 5.47% |
| Red-winged Blackbird (Agelaius phoeniceus) | 1.99% |
Most popular species is Humpback Whale (Megaptera novaeangliae).
We need your help! By making a contribution towards our hosting costs, you can help us provide an ad-free experience on our website in the near future. Paypal