Need two volunteers for beta-testing instructions for the upcoming Krita Bughunting event

Develop Artists Feedback & Testing
1

Hi, all!

As some of you might have heard we are planning a big bughunting event in September (right after releasing Krita 5.2.5). In the course of this event we want to join the efforts of the developer and the painters community to sort-out and eliminate as many bugs (and bugreports) in Krita as possible!

image
image482Ă—599 73.7 KB

Currently, Krita has too many bugreports on the bugzilla (over 950 reports!). Some of them are real bugs, some of them not, others are just really old bugreports for bugs that have been fixed long time ago.

In the course of the event we will have two teams: the testers team and the developers team. Testers will triage bugs and developers will fix them!

We would like to ask the painters community to join the testers team and help us sorting these bugs out: basically, one will have to look through some portion of the bug reports and decide if the bug is still reproducible or not. Or if the report can be updated and made more clear.

I have prepared detailed instructions for the testers team, but I’m not sure if it is clear enough and if there are any issues with that. Could you help me with that?

It would be nice if we could find two volunteers (one for Linux and one for Windows), who could spend a couple of hours on following the manual and trying to triage a few bugs. In the meantime I will try to help with the process, answer questions and adjust the manual :slight_smile:

How the bughunt process will work:

  1. You will read the instructions in the Google Docs
  2. Download a few testing packages for Krita (including Debug and ASAN versions, so it’ll be about 2 GiB of downloads)
  3. Then you will look through a few bugreports on bugzilla, test them and either CONFIRM or mark as NEEDSINFO

Later, developers team will take the bugreports you confirmed and will start fixing them! :slight_smile:

If you’d like to be a volunteer for testing the instructions, please reply in this thread! (or ask any questions! :slight_smile: )

6 Likes
2

I would like to volunteer as a tester for windows

3

I’m also on Windows but wouldn’t mind to test the instructions.

1 Like
4

Hi, @prutser and @ssenfukadavidjoshua!

Thank you for voluntreering to help! :slight_smile:

Here are the instructions document:

You need the part that is in the “Testing Team” chapter.

Just try following the instructions and triage a few bugs in a Krita component that is the most interesting for you. If you think that some instructions are not very clear or you have any troubles, write my here on on IRC :slight_smile:

PS:
Btw, the document is open for comments, so you can also ask/comment there directly!

PPS:

Btw, there are also two diagrams for the bug triaging process:

  • reproducible bug: link

  • non-reproducible bug: link

1 Like
5

All right! Just FYI (expectation management): my first opportunity to test this will be September 1st :slight_smile:

1 Like
6

Hi @dkazakov

I did a quick read of document, on my side it seems clear but I didn’t had the time to download/test AppImage with ASAN/Debug info.

There’re things I may miss or I’m wondering about:

  • Some reported bugs can be more technical than functional, how testing team should manage them? (risk: these kind of bugs will stay in REPORTED status…)
  • A bug opened by user X should not be tested/confirmed by the user X? :slight_smile:
  • You’re requesting 2 volunteers for beta-testing instructions, but concerning the bughunting event itself how it’ll be managed? (Who participate, when the bughunting event starts/ends, is there a “tester team leader” that will ensure triaged bug are properly triaged?)

On my side it will be unfortunately hard to engage my availability for the event: currently moving to a new apartment with a lot of boxes not opened yet and furniture not build yet, and in 3 weeks or less, there’ll be a big delivery here that will probably doesn’t let me time for anything… :face_with_peeking_eye::baby_bottle:

Anyway, I’ll try to follow what happen…

Grum999

2 Likes
8

I think it’s about having people review others’ reports with a fresh eye, otherwise I could go check my own reports and validate myself.
(I will gladly do so :laughing: )

9

That’s what I want to be sure: nobody will validate his own bug report :slight_smile:
But there’s no precision about that in the document

Grum999

2 Likes
10

I think it is not good to validate yourself if the version is same and your initial observation and the bug persists. Since you have already reported the bug you may not be adding any relevant information.

If you had reported the bug with older version and the bug is reproducible in the newer version I think a simple comment that the bug is still reproducible in newer version is okay.

if the bug is not reproducible anymore then that information is also welcome.

1 Like
11

Well you’re in luck, I happened to pick one that was filed by you :wink:

1 Like
12

All right, I’ve done a first round. I added some suggestions to the instructions document. I checked this bug report, so please have a look if I did what you expected :slight_smile:

Edit: and this one

13

Will you want help with the bug fixing as well, or only with triaging and testing? Less complex bugs could be assigned to volunteer developers.

14

Some reported bugs can be more technical than functional, how testing team should manage them? (risk: these kind of bugs will stay in REPORTED status…)

I don’t remember such bugs, except of the unittests bugs (and I guess we should just close them). Can you make an example of such non-functional bug?

A bug opened by user X should not be tested/confirmed by the user X?

Well, yes. The goal of the bug-triaging process is that the explanation of the bug is clear to at least one person who is not the reporter :slight_smile:

You’re requesting 2 volunteers for beta-testing instructions, but concerning the bughunting event itself how it’ll be managed?

Well, we’ll keep as many developers as possible for answering questions. I guess, I’ll have to manage it manually for the first time, just to gather some info.

Who participate,

From the KA community, anyone who wants to join. From the developers, everyone who is not busy with a personal project right now (I guess it’ll be about 4-5 people).

when the bughunting event starts/ends,

It is not decided yet. “After the release” and “for 3-4 weeks” is our target.

is there a “tester team leader” that will ensure triaged bug are properly triaged?

Having a tester team leader from the painters community would be a really good idea. Though we would need someone to step up for that. I wonder what should be the responsibilities of this person? I guess something like that:

  1. Manage the KA thread (and chat?) with the testers
  2. Assign “componends” and “bug groups” to different people (to make sure there are as little conflicts as possible)
  3. Verify the triaged bugs (at least randomly)
  4. Answer questions from testers (e.g. about bugzilla), and forward the difficult ones to developers

Theoretically, an experienced Krita user and bugreported could do that.

there’ll be a big delivery here that will probably doesn’t let me time for anything… :face_with_peeking_eye::baby_bottle:

Oh, congratulations! I wish you that everything goes smoothly and safely! :slight_smile:

15

Will you want help with the bug fixing as well, or only with triaging and testing?

Yeah, if you can fix bugs, you are welcome to join as a part of developers team :slight_smile:

16

Thank you, @prutser! That is exactly what I expected! :slight_smile: I have fixed a few of your comments in the document, replied to the others. I will try to reword the issues properly tomorrow.

1 Like
17

Hi

I can only talk about the one I’ve reported, but I suppose I’m not the only one to report bugs more from a technical point of view than from user’s point of view.

This kind of bug, where there’s strange message, not sure if there’s a real impact or not
https://bugs.kde.org/show_bug.cgi?id=488226
Some other bugs I’ve reported like this were already fixed :slight_smile:
But this kind of bugs (error message displayed on terminal) are more for “technical” user than for usual artists users (I’m probably not the only one to start my softwares from a terminal command line but we probably are a minority :sweat_smile:)

Or this kind of bugs, where description is easier (for me) to write from a technical point of view rather than trying to explain directly impact on Krita (there’s some explanation about real impact on Krita but for users it’s not easy to enter the edge case or easily see the impact because it’s not easy to understand without taking a look directly in technical part…)
https://bugs.kde.org/show_bug.cgi?id=470017

On an Open Source project based on volunteer work, I don’t have experience :slight_smile:

Ideally, yes these cases are under scope a tester team leader:

  1. Manage the KA thread (and chat?) with the testers
  2. Assign “componends” and “bug groups” to different people (to make sure there are as little conflicts as possible)
  3. Verify the triaged bugs (at least randomly)
  4. Answer questions from testers (e.g. about bugzilla), and forward the difficult ones to developers

But it can be difficult because people:

  • may not be in the same timezone
  • may be have a job or personal constraints aside and then not able to spent expected time on bugs (or just not available a day or two…)
  • are volunteers and may not be interested to work on some components or type of bugs (may be testers can inform the tester leader their preference about perimeter they prefer to test)

What can be done “easily” (but takes time)

  1. Track bugs that are taken in account by testers
    – Track who take a bug in charge, when
    – Ensure the bug taken in account is really taken in account (if after maybe 24h the bug is still not confirmed/unconfirmed, ensure the person who takes the bug is really working on it, if there’s some difficulties, or eventually “free” the bug to let someone else be affected on it)
    → A table in a dedicated thread in KA, or a Google doc (or Framacalc) spreadsheet can help

  2. Ensure consistency about bug reports
    – Similar bugs should be targeted the same way
    – Bug triaging must follow the rules that has been defined (for example, if a tester put a bug as critical, the team leader check if it’s really a critical or not)

  3. If a bug is confirmed, it should ensure that all required information are available for developer (detailed process to reproduce bug, screenshot if needed, .kra file if needed, version of Krita, OS, …)

  4. If a bug is unconfirmed, can check or ask someone else to check (according to environment -Linux, Windows- or understanding about bug report, may be the tester wasn’t in a situation to reproduce the bug) – take example here, even Halla was initially not able to reproduce the bug, so sometimes having different users testing can be helpful

  5. When a bug is fixed by a developer, ensure the tester who’ve confirmed the bug can confirm the bug is fixed (possibly, without regression aside…) or if tester is not available ask someone else to confirm the bug fix

  6. If there’s really difficult case or if not sure about a case, he ask a developer to take a look

  7. Ideally, manage a daily meeting (or one day on two)
    – With testers, to ensure everything is fine, if they’ve encountered difficulties with tools or process to reproduce bugs, share experience about specific case or method to facilitate tests
    – With developer(s) to avoid to disturb them at any time and ask them questions for bugs for which triaging is not clear (is it “Tools/Vector” or “Layers/Vector” for example?); also a short point about triaged bugs can be interesting (bug report clear enough? quality of triaging, what can be improved, …) and bug fix (priority to check fixed bugs, is it a sensitive bug fix - risk about regression?)
    → Not mandatory made through a video/chat meeting, it can be managed through a daily written report/questions - the idea is more to have once a day a centralized status about progress, experience, questions, …

It’s a full time job at the office (even if not exactly made like this)
I’m not sure if it can be applied as described here but the basics are described :slight_smile:
May be KDE team have some experience to share?

And yes, the one in charge should be someone with a good knowledge of Krita, if possible a minimal technical skills and of course a couple of hours of availability every day… (I’m thinking about @Michelist or @AhabGreybeard - sorry to give names :sweat_smile:)

Grum999

18

@Grum999, Sadly, nowadays I have far less free time than I used to have so I have that limitation.
Also, bug tracking/investigation/management is an intensive task that requires lots of concentration with lack of distraction over a period of time.
I’m not able to do that anymore :frowning:

19

@Grum999: Two years ago, I would have said yes immediately, or rather I would have joined the discussion in this topic from the very beginning, but there have been serious changes in my state of health that no longer allow this.

And sorry for this off-topic, but here are users with technical background reading along, so maybe they have a hint for me:
And my computer seems to have decided to do the same last night, at least I’ve been trying to reanimate it as far as possible for a few hours now, as it crashed severely for unknown reasons and also caused some data loss.
If anyone knows of a tool that can check the integrity of hard drives very quickly, I would be happy to receive a message via PM, as I need to check ~46TB, which takes ages with Windows’ on-board tools. :frowning:

Michelist

20

Well, that is a bug and have a way to reproduce, so that looks valid, even though technical :slight_smile:

Well, this migh be a bit too technical :slight_smile: But, yeah, given that there is a way to reproduce the issue and it causes bad user experience, it should be a valid bug.

It’s a full time job at the office (even if not exactly made like this)

That is exactly why I tried to write the manual in a way to make people triage bugs as autonomously as possible. So that we could avoid having a dedicated person giving out tasks maually. We have a bugzilla for that…