From Billboard Charts to Binary Arts: A Composer’s Dive into AI-Video
Artificial intelligence has found a new purpose in the music industry - visuals for music videos.
Musician Extraordinaire Don Harris has mastered the use of artificial intelligence to enhance his musical career and passion. Harriss’ latest video shares his “Spheres” excerpts.
I've been asked to describe my process for creating artificial intelligence-generated videos to accompany my original instrumental music. Here is a glimpse into my approach.
For many years, I've had the desire to pair my music with visual art. However, until a few years ago, this was a challenging and expensive endeavor—necessitating the hiring of animators and video effects artists. With the emergence of AI, generating visuals has become a task I can undertake on my own, though it does come with a bit of a learning curve.
First, I want to clarify that I do not use AI to compose my music. As a composer and recording artist, with Billboard Top Ten New Age CDs, the music excerpts I use are drawn from my existing repertoire. To hear more of my music, follow the link below to my Spotify page.
Now, let’s dive into how I create AI-generated music videos. The initial step in my process is choosing a piece of music I think will pair well with the visuals. Once I choose my music, I import a file into Adobe Audition, extract a segment, and make sure I have seamless edits and a coherent musical story arc. It’s important to note that the musical segment's length typically determines the duration of the video clip.
While selecting the music segment, I often start imaging the visuals to accompany the clip— be it a cool or warm color palette, specific landscapes like deserts, mountains or forests, or even alternative time periods, ranging from cyberpunk or steampunk to Victorian London or outer space. The music's essence heavily influences these choices, and the visual spectrum is boundless.
Once I have an inkling of the “look” I’m after, I open the Midjourney Discord app. Using their text-to-image process, I begin crafting a descriptive text prompt that will be used to create an image. Through a sometimes lengthy and iterative process of tweaking the text prompt, viewing results, and trying again and again, a useful image eventually emerges.
Once I’ve decided on the music and look, several possible paths on the creative road emerge. At this point, the challenge is to determine the best method for animating the chosen visuals. In the past, I've used a platform called Stable Diffusion for its img2img batch rendering capabilities and an extension named Deforum, which can produce videos based on image initiators and text prompts at varied time points.
My latest video release, titled "Spheres," leverages a new technique available in Midjourney 5.2. This feature allows for zooming out from a crafted or curated image, generating new content within the zoomed space. Panning left or right is also possible. Using this tool, I produce an image suitable for an extended "infinite" zoom. Segments of Deforum diffused video are also interspersed throughout.
Once I’m happy with the results, I use several different applications (which I'll link below) to edit and enhance the video. I usually assemble the various pieces of video in Adobe After Effects. During the AI process of creating video content, the images often become unstable after about five or 10 seconds, so it’s often necessary to create multiple segments before reaching a completed piece.
Blending original music with AI-generated visuals has truly opened up an exciting realm for me. While AI has streamlined the video creation process, the soul of the work remains deeply tied to the music. From choosing the right track to the final edits, the aim is a seamless fusion of audio and visual. For fellow artists, consider this a nudge to venture into the world of AI-enhanced creativity. Your next project could be a harmonious blend of your musical ideas and AI's visual capabilities. Dive in and let the journey begin!
Best,
Don for the Don’t Count Us Out Yet Team