CiptoHartanto

Background

In one of my recent projects, we wanted to use video background that plays based on how far the user has scrolled. I finally did it here on the home page of Finschia.

I found one really interesting article that talks about video scrubbing playback with JavaScript. The author, Stanko, mentioned in that post that what makes video scrubbing playback smooth is the key frame. In that article, he also mentioned a video converter called FFMPEG. The more key frames in a video, the smoother the experience can be.

Challenges

• In my case, the FFMPEG solution didn't seem to convert to .mp4 as expected since the converted videos were still sluggish. I didn't have lots of time to find out why since other tasks awaited for completion.
• .mp4 is sluggish on Firefox. For this type of background video, Firefox needs to be supplied with videos in .webm for smoother scrub playback. This also means there will be a condition to update when to display a.mp4 or .webm video.

Solutions

Video Conversion

After having quite a handful of trial-and-error situations, I recalled a software that I used to work with when I was a visual graphic design student. It's Adobe Media Encoder. I used to convert some video files for some subjects into other video formats before burning it onto a Compact Disc.

Luckily, I still have access to Adobe Media Encoder.

In the Adobe Media Encoder app, I updated a few settings as follows.

• Uncheck Export Audio because it won't be necessary and hopefully reduces the overall file size.
• Under Bitrate Settings, I find setting both Target Bitrate [Mbps] and Maximum Bitrate [Mbps] to be around 60 works. Please adjust as you wish.
• Under Advanced Settings, setting Key Frame Distance to be denser would make the results better. I chose 5.

Now, we are going to convert the .mp4 that was converted through Adobe Media Encoder through FFMPEG. You could follow along this tutorial on how to install FFMPEG on your machine.

I use the following command to convert the .mp4 to .webm:

Copy
1ffmpeg -i video-background.mp4 -c:v libvpx-vp9 -crf 5 -b:v 2500k -keyint_min 3 -g 10 -b:a 128k -c:a libopus video-background.webm
2

Browser detection to render proper video format.

As I mentioned earlier, proper video formats are required to get the butter smooth scrubbing playback effect.

These are the configurations:

• Firefox -> .webm.
• Other than Firefox -> .mp4.

One thing we need to remember is that the video file can't be loaded before we know which browser is being used. Once the video is loaded and playing, updating the video file won't give any effects.

Approaches

To update video timeline position, we can use video.currentTime, as explained in HTMLMediaElement: currentTime property. Since we need to know how far the user has scrolled, we'll couple it with a scroll event listener.

A scroll event listener that fires quite frequently may affect performance when a video is being updated. We can use requestAnimationFrames() to effectively update video.currentTime while the user scrolls through the page.

Another key factor that needs to put into consideration is how far the user has scrolled. In this example, I'd like to consider the document.body as the scroll area which will be the ratio of the final progress. The progress ranges from 0, which means the beginning of the video; 1 which means the end of the video.

Here is the code that only considers any browsers besides Firefox.

Copy
1import { useEffect, useRef } from "react";
2
3/*
4you can use this url of the original video file -- it's not converted yet
5expect to see sluggish scrubbing video playback
6const BACKGROUND_VIDEO_ORIGINAL = "https://videos.pexels.com/video-files/1943483/1943483-uhd_3840_2160_25fps.mp4";
7*/
8
9export default function VideoBackground() {
10  const refVideo = useRef<HTMLVideoElement | null>(null);
11
12  useEffect(() => {
13
14    const scrubVideoPlayback = () => {
15      const { scrollHeight } = document.body;
16      const offset = window.scrollY;
17
18      // scrollProgress is from 0 through 1
19      const scrollProgress = Number(
20        Number(offset / (scrollHeight - offset / 3)).toFixed(3),
21      );
22 
23      // scrubVIdeo runs when scrubVideoPlayback() is running
24      const scrubVideo = requestAnimationFrame(() => {
25        if (refVideo.current !== null) {
26          const video = refVideo.current;
27          const videoDuration = video.duration;
28
29          //  scrollProgress value will equalize what the actual video timeline should be
30          video.currentTime = videoDuration * scrollProgress;
31        }
32      });
33
34      // when video is finished, stop scrubVideo()
35      if (scrollProgress > 1) cancelAnimationFrame(scrubVideo);
36    };
37    window.addEventListener("scroll", scrubVideoPlayback);
38
39    return () => window.removeEventListener("scroll", scrubVideoPlayback);
40  }, []);
41
42  return (
43    <div className="video">
44      <video className="video-tag" playsInline muted ref={refVideo}>
45        <source src="video-background.mp4" type="video/mp4" />
46      </video>
47    </div>
48  );
49}
50

As I mentioned earlier, we need to distinguish if the browser is Firefox or not. Through using useState, we can set an initial value of isFirefox to be undefined. The reason behind this is to make sure which browser is actually being used after the first render. In the first render, the result is always undefined, which returns an empty markup. We expect to rerun <VIdeoBackground /> until isFirefox has a boolean value. The following is how we could approach that.

Copy
1import { useEffect, useRef, useState } from "react";
2export default function VideoBackground() {
3    const [isFirefox, setIsFirefox] = useState<undefined | boolean>(undefined);
4    ...
5    useEffect(() => {
6      const isBrowserFirefox = navigator.userAgent.indexOf("Firefox") !== -1;
7      setIsFirefox(isBrowserFirefox);
8    }, []);
9
10    if (isFirefox === undefined) return <></>;
11    return (
12    <div className="video">
13      <video className="video-tag" playsInline muted ref={refVideo}>
14        {isFirefox ? (
15          <source
16            src="video-background.webm"
17            type="video/webm"
18          />
19        ) : (
20          <source src="video-background.mp4" type="video/mp4" />
21        )}
22      </video>
23    </div>
24  );
25}
26

End Results

Demo Links

• Preview
• CodeSandbox

Here's the final code:

Copy
1// components/VIdeoBackground.tsx
2
3import { useEffect, useRef, useState } from "react";
4
5export default function VideoBackground() {
6  const [isFirefox, setIsFirefox] = useState<undefined | boolean>(undefined);
7  const refVideo = useRef<HTMLVideoElement | null>(null);
8  useEffect(() => {
9    const scrubVideoPlayback = () => {
10      const { scrollHeight } = document.body;
11      const offset = window.scrollY;
12      const scrollProgress = Number(
13        Number(offset / (scrollHeight - offset / 3)).toFixed(3),
14      );
15
16      const scrubVideo = requestAnimationFrame(() => {
17        if (refVideo.current !== null) {
18          const video = refVideo.current;
19          const videoDuration = video.duration;
20
21          video.currentTime = videoDuration * scrollProgress;
22        }
23      });
24      if (scrollProgress > 1) cancelAnimationFrame(scrubVideo);
25    };
26    window.addEventListener("scroll", scrubVideoPlayback);
27
28    return () => window.removeEventListener("scroll", scrubVideoPlayback);
29  }, []);
30
31  useEffect(() => {
32    const isBrowserFirefox = navigator.userAgent.indexOf("Firefox") !== -1;
33    setIsFirefox(isBrowserFirefox);
34  }, []);
35
36  if (isFirefox === undefined) return <></>;
37
38  return (
39    <div className="video">
40      <video className="video-tag" playsInline muted ref={refVideo}>
41        {isFirefox ? (
42          <source
43            src="video-bacgkround.webm"
44            type="video/webm"
45          />
46        ) : (
47          <source src="video-bacgkround.mp4" type="video/mp4" />
48        )}
49      </video>
50    </div>
51  );
52}
53

Copy
1// App.css
2.wrapper {
3  width: 100%;
4  height: 500vh;
5  display: flex;
6}
7
8.video {
9  position: fixed;
10  top: 0;
11  right: 0;
12  bottom: 0;
13  left: 0;
14  width: 100%;
15  height: 100%;
16  overflow: hidden;
17}
18
19.video-tag {
20  position: absolute;
21  height: 100%;
22  top: 50%;
23  left: 50%;
24  transform: translateX(-50%) translateY(-50%);
25}
26

Copy
1// App.tsx
2import "./App.css";
3
4import VideoBackground from "./components/VideoBackgorund";
5
6function App() {
7  return (
8    <div className="wrapper">
9      <VideoBackground />
10    </div>
11  );
12}
13
14export default App;
15

So, the keys to butter smooth scrubbing video playback are:

• Dense key frame distance which helps in getting a precise image in the video. Utilizing FFMPEG to convert videos can be one of the free options.
• Provide at least two video formats, ,mp4 and .webm. Play .webm vdeo on Firefox browsers and .mp4 for the rest of the browsers (Chrome, Safari, etc.).
• Attach a scroll event listener with requestAnimationFrame() to update video.currentTime. To compute scrollProgress, we can do it manually or consider Framer Motion - useScroll. With Framer Motion, the scroll reference can be anything, not only <body />.

Considerations

• Visually, video backgrounds look really cool. However, for a sharp video file, it could be over 10MB in size. As in my solution, there should be two video formats to ensure scrubbing video playback works in all browsers as expected. This may add extra loads to your website.
• Adding smooth scroll plugin to the project that will add inertia to the scroll action to leverage the scrolling experience.
• The proposed methods may not work with all videos since each video is generated differently to the one in the example above. If you have access to the video producer, please inform them about the special requirements.