Enhancements on Recorder plugin

Version 5.0 brings a great Recorder plugin, I wanted it since I first started drawing in Krita. Unfortunately it seems to be on early development stage and not suitable for production use.

What it lacks:

  1. No automatic recording. You have to press the record button when you start drawing and stop when you finished.

  2. No noticeable visual indicator like “Recording in process” - only small button which is pressed… or not?

  3. Freeze for seconds after stopping recording - encoding the video? I have 8 seconds of freeze for very simple test painting. I wonder how long it would be for the real 7-12 hour session? Do record get lost on crash?

  4. Works only on Linux due to ffmpeg lib dependency - not a real problem for me, but what about other users?

  5. Cannot control quality, resolution nor timelapse speed.

Some thoughts on improvements:

  1. Make recording fully automatic by moving recording settings to “Configure Krita”. You can enable automatic recording of timelapse there and configure something like output directory and prefix for timelapses. Also it’s a good idea to use document name as the video name prefix, maybe settings to enable recording to separate directories. When auto recording is active in settings it automatically starts recording when user creates a new document or open existing and edit it. It stops when user closes document or Krita.

  2. Add separate toolbar with recording controls like red recording button, :red_circle:. Button is pressed(checked) when recording is active, user can temporary disable recording by pressing (unchecking) it. Export video button - join all the parts to the single video and let user choose where to save it (Like Render Animation…). Menu items in Tools -> Recorder -> Start/Stop, File -> Export Recorder Timelapse…

3 & 4. Make it much faster and make it available on all currently supported platforms by replacing ffmpeg to something simpler like MJPEG. By my experience it has very low CPU load and pretty reliable - it writes just like a sequence of JPEGs to the single file. Also it shouldn’t need any additional encoding libraries. The instant start & stop will be available as well - no encoding after stop needed.

  1. Make settings for:
    • quality (JPEG compression)
    • resolution: original, 1/2, 1/3, 1/4 (maybe?)
    • timelapse speed/interval of taking snapshots.

Any other thoughts about this plugin?

I can try to implement these improvements, but it may take some time (because I’m not so experience in Krita codebase - only submitted few patches). Or maybe someone other want to do it…?

5 Likes

Thank you for the feedback. The recorder plugin was implemented by a first time contributor and that too they had it done for themselves and later up streamed it.

I also wanted the recording feature so I am happy that it got upstreamed.

I am a bit sceptical about auto recording, since it will have some performance penalty. So my personal opinion is to have it in the docker and may there you can have auto recording. The recording just captures/exports your artwork at fixed intervals (1s) if you are drawing on the canvas. So I think auto recording will definitely have some quirks. If there is no performance hit then it will be cool.

I agree that it is now not clear whether recording is active or not. This was the feedback from initial testing as well. May be we can improve the UX of the docker. I also wished for more controls like choosing file format to save etc.

The time it requires to compile and export the webm can be better highlighted by a progress bar.

Regarding MJPEG, is it patent encumbered like mpeg? If yes then it is not ideal.

If you could improve this plugin that is really great.

1 Like

Well, I don’t talk about always-on automatic recording. It can be enabled from settings and it’s disabled by default. You can still use docker to start and stop it manually.

I’d like to allow user to configure the capture interval, so it would be possible to get a balance between quality and performance. Also to control the speed of timelapse. It will be encoded in separate thread so there shouldn’t be any noticeable performance impact.

Any thoughts about activity indicator? I’m thinking about adding something like “:red_circle: Recording…” to the status bar near memory usage indicator.

MJPEG is just a sequence of JPEGs joined into single file. They just go one after one, no codec nor special support needed. You can even play it in your browser. To convert it to the more convenient video format joining all the parts external ffmpeg can be used like for exporting animation.

:slight_smile:

Looking one of specifications, it’s not just a sequence of JPEG file joined in a single file…
And as precised on Wikipedia it seems that there’s no really official specification, each manufacturer doing it like it want…

Unlike the video formats specified in international standards such as MPEG-2 and the format specified in the JPEG still-picture coding standard, there is no document that defines a single exact format that is universally recognized as a complete specification of “Motion JPEG” for use in all contexts. This raises compatibility concerns about file outputs from different manufacturers. However, each particular file format usually has some standard on how M-JPEG is encoded. For example, Microsoft documents their standard format to store M-JPEG in AVI files,[5] Apple documents how M-JPEG is stored in QuickTime files, RFC 2435 describes how M-JPEG is implemented in an RTP stream, and an M-JPEG CodecID is planned for the Matroska file format.[6]

And M-JPEG is not an efficient video file format…

I think that’s maybe a better choice in this case to save a sequence of JPEG/PNG files directly in a directory and then, let use your favorite encoder/video maker to create a video from sequence :wink:

Grum999

Oh, yes, you’re right. Thanks for providing info. I remember we used sequence of JPEGs to record some “video” on low-end embedded device joining it together. There was a tool to play the resulting file.

Hmm, but I don’t think saving a lot of JPEGs as separate files is the way to go. If you draw for 12 hours it would be 12×60×60=43200 JPEGs and slow when accessing the directory. Probably it would be better to save it into a single (per session) data file in specific format (ex: size (4bytes), jpeg data, …).
After that external ffmpeg can be used to convert it to video. To avoid extraction of single JPEGs from data file there is image2pipe option – need to try it.

And the time of encoding PNG may be longer to a level of magnitude, so I’d stick with JPEG.

If you want to capture images during 12h at 1fps, yes, it will generate a lot of images and yes, your explorer (linux or windows) may a little bit suffering to display you directory’s files list :dizzy_face:

Yes, writing a proprietary container to store JPEG is easy.
But after you’ll need a dedicated tool to convert it to a known file format and then, much work to do…

On my side, using ffmpeg (at 1fps) doesn’t consume too much CPU:
ffmpeg -video_size 1920x1080 -framerate 1 -f x11grab -i :2.0 screen_output.mp4
Around 4~6% CPU on one CPU.

Ok, that’s not integrated into Krita but maybe everything don’t have to be integrated into Krita…? :thinking:

There’s very efficient tools designed to do specific tasks, especially for video: my point of view is to let them working properly aside Krita instead of trying to implement very specific functionalities that Krita is not aimed to manage (and then, let Krita being improved for what he’s really designed to :wink:)

For example, I use Krita’s rendering as video only for checking animated sequence.
When animation is finished, I do export as a PNG file sequence, and I use dedicated video editor tools to finalize video.

Grum999

Well I didn’t think about making it readable by user, is this really needed? Maybe it’s better allow user to export it like in “Render animation” dialog… If user wants JPEGs, let them get it as JPEGs :laughing:.

I used screen recording for years and the recorded timelapses may cause epileptic seizure :sweat_smile: because I use zoom and rotate tools too much… Don’t recommend anyone to watch it, especially after lunch :sweat_smile:.

Seriously, many of professional painting app has builtin recorder, ex: Procreate, even mobile app Ibis Paint X has it! It records automatically and doesn’t lag.

Oh ok, just export the canvas always at 100% size, whatever the current zoom is…
I Understand :sweat_smile:

You need to read your file to produce a final video you want to share, no? :grimacing:

So if you create a dedicated container, finally you’ll have to provide a tool to read it and export the saved content to build the final video (ie: generate JPEG files in a directory to let ffmpeg -or any other video encoder tool- generate a video from sequence)

After, in this process, the JPEG files are extracted in a temporary directory and deleted once video is made.
You normally don’t have to explore the directory and it should be Ok to have thousand of files in the directory :slight_smile:

Grum999

I planned to make this “tool” as a part of Krita, like animation export. Do Krita have a separate tool to export animation from KRA?

No, as I said before, I plan to pass them to ffmpeg using pipe, no temporary files created. I didn’t yet tried the image2pipe option yet, but it should be possible.

I’m not sure, creating a new file in that dir will take longer and longer time… Well, I don’t mind trying out this, but I think it’s just ineffective.

Yes all the settings can be in the docker itself rather than scattered around two places. I agree with your point of having more options like specifying time interval and making the record action more prominent etc.

I actually tested this feature in production, with the time limit of 1s and I painted for 3 days pausing in between. the work may be of 24hrs total. The webm that it produced was of the resolution of the original image so I agree that there can be some options to downsize and options for setting quality.

In my observation even in 1s time interval many of my strokes and actions were skipped since we can paint more than 1 stroke per second. so I think increasing the time interval can be very jarring for the timelapse, in any case it depends on the choice of the artists to set the time interval.

Yes since it was in demand. And fortunately some volunteer stepped forward to implement in Krita. We can improve it further. Krita uses ffmpeg for rendering animation. So it was efficient and good choice to use ffmpeg for this task too. I think @emmetpdx was working on getting only important bits of ffmpeg to work. And I agree with @Grum999 we should utilise tools which are best for certain tasks. Mjpeg might be good but will it be easy to maintain in the long run? at least ffmpeg will be around and krita can ofload this task to it. Some things might seem easy to do but in reality it requires effort to code and also maintain it later.

Even in this particular recorder there are corner cases that need testing like what happens when you have two window which are being recorded. What happens when autosave is triggered at the moment you stop recording and the file is being compiled. I am just stating some obvious ones I am sure there will be others.

Hope more people get inspired and improve this feature.

3 Likes

Hi, here is some update.

I partially implemented the enhancements. For current development status please see this MR: https://invent.kde.org/graphics/krita/-/merge_requests/522


Here going some screenshots:

Configuration options, recording not started:
image

Recording started, Active REC indicator, using document name to record timelapses for 2+ documents:

“Paused” indicator (Idle - no changes since writing the last frame):
image

1 Like

Have you seen my tutorial here: Image recording Procreate style (no zoom/rotate!) - but better (shows strokes during making them)! - it allows you to make a no-zoom-or-rotate style of timelapses with any window recorder (not every screen recorder because it needs to record a particular window).

2 Likes

Another enhancement I suggested to the initial pluginw as to remember the folder where the video/ files are stored. Often we may pause close krita and come back to working on the artwork next day . At that time ti becomes cumbersome to again input the save location.

Thank you for working on enhancing this. If you want to discuss about ffmpeg please join IRC for real time chat.

Yes, but this is too complicated for cats like me :laughing:… I’d like to single push a button (or don’t push it at all) to make a recording of my speedpaint.

Storing of recordings settings is already implemented. So when you restart Krita you will have exactly the same settings (folder, interval, etc…) like the last time you changed it.

1 Like

I recorded a short timelapse using current version of plugin (used ffmpeg from command line to convert jpegs to mp4)

Disabling image locking seems to produce noticeable artifacts (presented after second part in video), but enabling it breaks selection dependent tools like Move and Transform. Need to think how to do it better, maybe pause recording on activating that kind of tools…

1 Like

I think it works really good.

The recording you have done is very smooth compared to my recording done with the existing plugin. I will test your branch too. How much interval you have set for your recording?

May I ask what would be the benefits of using a recording plugin over something like OBS? Maybe ease of use since I can see it being streamlined and simplified for a more specific purpose(s).

1 Like

If you record timelapses with external tools (OBS, EasyScreenCast, etc…) you have to:

  1. start and stop recording manually, if you forgot to stop it will bloat your hdd, if you forgot to start - you’ll get a missed part
  2. don’t zoom/rotate canvas to much else everyone who watch your timelapse will feel dizzy
  3. control timelapse options somehow, till now I used ffmpeg command line to create 30fps timelapse from 1fps realtime video, not very convenient.

There are few features in Recorder:

  1. You can record timelapse without using external tools
  2. You just have to press one button to start/start stop record
  3. “Record automatically” option - if you enable it all your timelapses will be recorded automatically, and you don’t need to start/stop manually
  4. It records only when canvas changes, so if you started drawing with record enabled and gone drinking tea for a hour or two, it doesn’t fill your HDD by still video. This works even if you started it by pressing “Record” button
  5. Canvas is always recorded in fixed orientation, so if you use tools like zoom or canvas rotation it will not affect the timelapse.
  6. Convenient settings on resolution, FPS and so on
  7. After finishing your drawing you get your timelapse just by clicking “Export” (optionally setting FPS and resolution) and “Save”.
2 Likes