Fighting with new improved "resource system"

Develop Artists Feedback & Testing
1

Hi, I enjoy the fact fifth KRITA was born, but… have some complaining about new
or improved (?) resource system.
As @tiar wrote to me in other thread:
Easiest way is to go to Settings - Resource Manager - and then select that bundle, use the “All” tag, select all brushes and use the little ‘+’ button to add a category. Btw that category is called a “tag” (you’ll have an easier time looking through tutorials etc. knowing the terminology, that’s why I correct it).
I’m not sure whethrr it should add a category automatically nowadays or not… maybe not… but it’s easy to fix, at least.

Well, this works… almost or “so so”. I am not fully satisfied with it, to be honest. :frowning:
Nearly every one bundle I installed recently, is not visible (in its “tag” name" at upper sub-window) and seems to be hardly fixable. :frowning:
For example, I wanted to have @Rakurri brushes v2 properly installed in my KRITA v5.02.
I installed the bundle but it didn’t show up in the list of brushes sets.
When I try to fix it like Tiar adviced above, it didn’t work at all (no reaction and no results). :frowning:
What is getting worse, when I try to set completely new “marking” to this brush, Krita hangs completely in few seconds or more.

I have KRITA v5.02 on updated Windows 10 64-bit (32 GB Ram, i7-6700K processor, etc.)
What is wrong? :o:

Here is the small video about it:

2 Likes
2

In the video I see you tag lots of brush at the same time and it hang for some second.
Yesterday I was doing the same thing, tagging lot of brush tips at the same time, and noticed that when krita hanging, my HDD is in heavy load.

3

@Lesqwe56 Thx for comments.
Yeah, my HDD is also under heavy pressure but in most cases KRITA hangs so bad that it needs to be terminated in system. :skull: :persevere:

There must be seriously something “spoiled” or still bugged as even some creators fed up with it and still (!) prefer to use “hand method” when installing their resources. For example, here it is:

It makes me really sad and think something is still bad done in Krita’s resource-system, unfortunately.

4

Unfortunately Rakurri’s brushes are somewhat heavy, maybe labeling them in small groups helps.

5

@SchrodingerCat
But @Rakurri s brushes are only the example! I have some similar situations.

Should user has to manually fix every brush, one-by-one as @tiar likely suggested at the start ? It would make this so loooong time. :frowning:

6

I manage to tag Rakurri v2.0 brush in my laptop [16gb ram] and PC [64] slower in laptop but manageable [if you have patience to ignore it].

What I did to figure out all the brush is go to “All Un tagged” then switch the resource to Rakurri V2.0.

Tagging so many resources - bog down the system, if they are heavy more so. I sit out and watch youtube. Eventually it finish.

Tha lag is noticeable even in my PC [there’s still slight noticeable lag] but not as crazy as in my laptop but yeah if you sit out do something it gets done [frustrating still] and badly need of optimization.

2 Likes
7

Hi

Made some tests

Import bundle take times (20-30s maybe).
The only “boring” thing is lack of activity information (a progress bar, even in “infinite mode”, could be useful :slight_smile: )

But selecting all brushes and tagging, that’s very weird: my Linux session is totally freezed.
While tags are applied, I can see 1 CPU running a 100% but my system is just unable to switch to another application, OS is totally unresponsive :confused:
And it take times.

Looking what happen on sqlite database, tags create few records (143)
Nothing that can justify this for so few records inserted in table “resource_tags”

I took a quick look with some console print

Creating and affecting a new tag to all of 143 brushes generate:

  • 1 call to KisWdgTagSelectionControllerOneResource::slotCreateNewTag()
    which generate (from here):
    → 2145 calls to KisAllTagResourceModel::tagResource() (FILE wdgtagselection.cpp)
    then 15 calls per brush…??

  • From the 2143 calls, 143 call execute to KisAllTagResourceModel::resetQuery(); this is long; a “big” query database to refresh model…

  • There also additional SELECT statement that can be optimized…
    KisAllTagResourceModel::isResourceTagged() is executed here 2145 times for example, on a table for which there’s no index on accessed keys

Also, it seems calls are made outside sql transaction (but need to check deeper) then it’s slower than if made within a transaction (but ok, the real problem for me is the refresh of model)

Didn’t check but:

I think a formal bug might be created.

Grum999

6 Likes
8

I didn’t suggest one by one though, in Resource Manager you can select all of them at once.

Oh I see that’s where the confusion comes, you did that, it just didn’t work fast enough, so you thought it didn’t work at all. And @Grum999 seems to have an idea why it was like that.

I must admit I never tried with so many bundles and so many presets…

9

Some additional test…

Execution time takes between 194 and 240s on my side…
the resetQuery() is long: 0.8 to 0.9s to retrive ~1300rows

There’s a GROUP BY clause without any aggregation functions.
Not sure why.
Removing the GROUP BY, the resetQuery() execution time is around 0.0007s
For a total execution time reduced to 30s

Still not good, but better

The resetQuery() might be called only once time.
Also need to find a solution to call tags udpate only 143 times would really improve this dialog box :slight_smile:

Another thing, didn’t took a look yet: juste doing a selection of all brushes took about 15-20s…
There’s might be multiples call made somewhere

Grum999

1 Like
10

Just for test, removing the resetQuery() from KisAllTagResourceModel::tagResource() reduce total execution time to 0.37s :slight_smile:

I’ve difficulties to understand everything in code, and not really the time to dig more, but global improvement axis:

  1. need to understand why there’s 15 call per brush to update one tag
  2. need to resetQuery() only once (when all tags are updated), with a signal or maybe a flag “updateInProgress” to avoid resetQuery() being executed in this case
  3. need to improve SQL SELECT statement resetQuery()

For testing, load import all possible bundles and start to create tags and affect them to presets, could give a good indicator about performances :slight_smile:

Grum999

3 Likes
11

I have something else planned this week… :rofl:

Michelist …again Off-Topic

1 Like
12

Uhh, that seems like a task for me, damn :frowning:

Thank you very much for the analysis.

Can you please tell me if I saw this effect with a standard, default number of bundles, or should I import a plenty of them like in the video?

Oh and btw, if someone makes a bug report, please let me know.

4 Likes
13

Can you please explain what you mean by “without any aggregation functions”? I guess the meaning of this GROUP BY was to deduplicate duplicated (in different storages, but the same name and md5, which means the same content) resources. The previous solution to just mark the newly coming resources as inactive was not working well.

14

It’s visible, but less than with additional bundles.

test Default installation (59 presets) Tested (412 presets + 40tags)
Select all brushes in listview no perceptible delay 15-20s delay
create new tag on all Krita_4_default resource presets 3-4s delay 550s delay

Time is correlated with total number of preset + number of tags

A GROUP BY clause is usually used to indicate how to aggregate data.
Example:
SELECT a, b, sum(c) FROM t GROUP BY a,b

But
SELECT a, b, d, sum(c) FROM t GROUP BY a,b
or
SELECT a, b, d FROM t GROUP BY a,b

You don’t know what the db engine will return.

For example, this type of query in an Oracle database can’t work because when a GROUP BY is used, Oracle expect to get an aggregation (COUNT, SUM, AVG, …) and only column from GROUP BY are expected in non-aggregated SELECT columns

SQLite don’t have the same “limitation” than oracle engine, that’s sure.
But here, when you do that, you return one line on given group key, but you don’t know which one.

Also clearly, the execution time gap is so huge between query with/without GROUP BY that for me means that SQLite don’t manage it properly.

I’ll try to take a look closer to this query, how it could be improved.

Currently, that’s anormaly too long:
With group by

image
image1920×1028 218 KB

Without group by
image
image1920×1028 214 KB

7 “duplicated” rows removed with group by clause; but looking in detail it seems these rows are from different storages.
Don’t know if it’s normal or not, and which one really have to be used nether if it’s important or not :man_shrugging:

I’ll try tonight to review this query, if it can be improved

Grum999

2 Likes
15

Incidentally, I was trying to bite this situation from another angle and started to change tagResource() function to take multiple resource ids at once. And got to the same monstrum of a sql query :wink:

I’m pretty sure I’ll need to run something that would be 100% sure to return me the exact same rows plus one the same but uncluding inactive tags/resources… (who added the “active” column there anyway?).

I think it should just use SELECT DISTINCT, but see this: Fix resource duplication when any tag is selected (e23a445d) · Commits · Graphics / Krita · GitLab

16

So what to do now? What I am supposed to do ? Isn’t anything broken about that resource-system you worked in for some years??? :open_mouth:

BTW. today I got to know @Soma declared he will revert to Krita v4 which is far much stable, predictable and fast… What a pity…

17

no a select distinct probably wont return the same result

select distinct a, b, c, d from t
will return distinct rows on all a,b,c,d columns

while here
select a, b, c, d from t group by a,b
will return distinct rows on a,b columns

the thing is, you can’t predict which c,d value you’ll retrieve

can be different on each query results…
and that’s really disturbing for me.

Grum999

18
  1. you can wait for a bug fix and do with it, but can’t tell how long time you’ll have to wait for it
  2. you can do a rollback to previous version if problem is too important for you

Also, there was 5 beta
nobody return any problem I think about this specific problem during beta… :man_shrugging:
That’s easy to complain now, but developpers need more than negatives feedback about their work
Some positive support despite the problems encountered would be more appropriate and encouraging for them :blush:

Grum999

6 Likes
19

I think I have seen this often, many times beta are not used by everyone, thinking that it will be buggy. Of course it will be buggy that is why it is a beta and beta is there to find the bug so that it doesn’t go into stable. But I don’t blame people for not beta testing also it is not right to do that. Although I wish more people would do it.

20

Yes, I thought similarly. But I guess I saw some bug-reports about resources’ system.
And what I understand when v5 finally appeared, people have been told it was “built from ground up” or something like that. But now it seems it works kinda worse than before.