Simulating Mouse Click on Canvas to Use Fill-and-Enclose Tool

[Citrus8684]

Hey! So after hearing that you can’t really use the Enclose-and-Fill tool via the API a few days ago, I’ve been wracking my brain trying to think of some other way I could perform my task, but everything else is just too inconsistent.

To come to the point, I want my plugin to use the Fill-and-Enclose tool over the entire canvas by simulating mouse clicks. Is this possible, and if yes, I’d greatly appreciate a few pointers.

Thanks in advance!

My exact usecase and failed ideas

What I want to do, is create a simple and fast way to paint over all text in a bubble:

How I went about this in Clip Studio Paint, was by selecting the white in the bubbles with the Contiguous Selection tool (with a few px of shrink), filling the area with white, then running a special fill (with the “fill enclosed” option) over the entire canvas. However, in Krita, as my plugin can’t easily make use the Fill-and-Enclose tool in any way, this isn’t as simple as I had hoped.

One Idea I had was to select the white areas of the bubbles, then grow and subsequently shrink the selection, and fill it.
The problem with this method is, it oftentimes misses characters if the selection wasn’t grown enough, and if bubbles are close together/the selection was grown too much, the selections often merge during the grow step, and won’t separate again when shrinking. This results in unintended parts of the image being filled in.

Another idea was to select the white, fill, deselect, then select the transparent area by clicking somewhere outside of the bubbles. Finally, invert and fill again.
But this would again require to simulate a mouse click.

[Celes]

I don’t have time to try it out in Krita right now but yes, if it responds to simulated mouse events then it should work. You’d have to either create:

1. Button press (QMouseEvent)
2. A number of mouse moves to enclose the area (QMouseMove)
3. Button release (QMouseEvent)

Or again do a press and release but for movement set the cursor position instead.

A few links that might be useful:

Simulate a mouse release Pyqt

SImulate a mouse move in pyqt

This one is C++ Qt and badly formatted but can be translated to Python:

Simulating mouse events - properly [Solved]

[Grum999]

Hi

You can take a look here:

Clicking with scripts

Hackish way to generate QTableEvents. Create new document and run script in Scripter. Note: pan, rotate and zoom are ignored. from random import random from krita import Krita from PyQt5.QtCore import ( Qt, QPoint, QPointF, QEvent) from PyQt5.QtGui import ( QTabletEvent,) from PyQt5.QtWidgets import ( QWidget, QApplication) def find_current_canvas(): app = Krita.instance() q_window = app.activeWindow().qwindow() q_stacke…

Grum999

[Citrus8684]

@Grum999 @Celes
Thank you both! I’ll see if I can throw something together that works here and close the topic if it does.

[Citrus8684]

All right, so by inverting one of AkiR’s other snippets that he posted here, I managed to make a function that clicks roughly at the intended document coordinates, regardless of zoom etc.

from krita import Krita
from PyQt5.QtCore import *
from PyQt5.QtGui import *
from PyQt5.QtWidgets import *

def get_q_view(view):
    window = view.window()
    q_window = window.qwindow()
    q_stacked_widget = q_window.centralWidget()
    q_mdi_area = q_stacked_widget.findChild(QMdiArea)
    for v, q_mdi_view in zip(window.views(), q_mdi_area.subWindowList()):
        if v == view:
            return q_mdi_view.widget()


def get_q_canvas(q_view):
    scroll_area = q_view.findChild(QAbstractScrollArea)
    viewport = scroll_area.viewport()
    for child in viewport.children():
        cls_name = child.metaObject().className()
        if cls_name.startswith('Kis') and ('Canvas' in cls_name):
            return child


def get_transform(view):
    def _offset(scroller):
        mid = (scroller.minimum() + scroller.maximum()) / 2.0
        return -(scroller.value() - mid)
    canvas = view.canvas()
    document = view.document()
    q_view = get_q_view(view)
    area = q_view.findChild(QAbstractScrollArea)
    zoom = (canvas.zoomLevel() * 72.0) / document.resolution()
    transform = QTransform()
    transform.translate(
            _offset(area.horizontalScrollBar()),
            _offset(area.verticalScrollBar()))
    transform.rotate(canvas.rotation())
    transform.scale(zoom, zoom)
    return transform

def get_global_from_document_coords(doc_x, doc_y):
    app = Krita.instance()
    view = app.activeWindow().activeView()
    document = view.document()
    if document:
        q_view = get_q_view(view)
        q_canvas = get_q_canvas(q_view)
        transform = get_transform(view)
        
        # Calculate the center of the document
        centerCanvas = QPointF(0.5 * document.width(), 0.5 * document.height())
        
        #Convert document coordinates to coordinates relative to the center
        local_pos = QPointF(doc_x, doc_y) - centerCanvas
        
        # Apply the transformation to convert to canvas coordinates
        transformed_pos = transform.map(local_pos)
        
        # Find the center of the canvas
        center = q_canvas.rect().center()
        
        # Convert canvas coordinates to local widget coordinates
        widget_local_pos = transformed_pos + QPointF(center)
        
        return QPoint(int(widget_local_pos.x()), int(widget_local_pos.y()))



def click_canvas(x0, y0):
    pos0 = get_global_from_document_coords(x0, y0)
    app = Krita.instance()
    view = app.activeWindow().activeView()
    document = view.document()
    if document is None:
        return

    q_view = get_q_view(view)
    canvas = get_q_canvas(q_view)    

    global_pos = QPointF(canvas.mapToGlobal(pos0))
    device = QTabletEvent.Stylus
    pointer_type = QTabletEvent.Pen
    pressure = 1
    x_tilt = 0
    y_tilt = 0
    tangential_pressure = 0.0
    rotation = 0.0
    z_pos = 0
    key_state = Qt.NoModifier
    unique_id = 1234  # ???
    button = Qt.LeftButton
    buttons = Qt.LeftButton

    canvas.activateWindow()
    
    table_press = QTabletEvent(
        QEvent.TabletPress,
        pos0,
        global_pos,
        device,
        pointer_type,
        pressure,
        x_tilt,
        y_tilt,
        tangential_pressure,
        rotation,
        z_pos,
        key_state,
        unique_id,
        button,
        buttons)

    table_release = QTabletEvent(
        QEvent.TabletRelease,
        pos0,
        global_pos,
        device,
        pointer_type,
        pressure,
        x_tilt,
        y_tilt,
        tangential_pressure,
        rotation,
        z_pos,
        key_state,
        unique_id,
        button,
        buttons)

    QApplication.sendEvent(canvas, table_press)
    QApplication.sendEvent(canvas, table_release)

app = Krita.instance()
view = app.activeWindow().activeView()
document = view.document()



click_canvas(100,100)

Krita.instance().activeDocument().refreshProjection()

However, just clicking is unfortunately not good enough. I’ve got the Enclose-and-Fill tool in Rectangle mode, so I’d like to drag it from 0,0 (pos0) to the max width and height of the document (pos1).

Setting the release coordinates to pos1 doesn’t work, even with throwing in a few hacky sleep statements that I stole from somewhere:

def sleep(value):
    loop = QEventLoop()
    QTimer.singleShot(value, loop.quit)
    loop.exec()

Similarly, clicking down in the four corners in lasso mode before releasing doesn’t work either.

Does anyone have an idea how I could make it properly drag accross the area?
If no, I guess I could switch the tool to brush mode, and just click across the entire canvas, but that’s a much more unreliable solution than dragging accross the entire canvas, and I’d have to figure out a way to set the pen size.

[Deif_Lou]

Try making a tablet move event identical to the release one but with the type changed to QEvent.TabletMove, and then send it between the ohher 2 events.
Edit: wait, it seems that you use the same position for the press and release events in the code. You should use pos0 in the press event and pos1 in the move and release events.

[Citrus8684]

Thanks for the suggestion, and sorry for not being more clear. The code I posted above was just what I got working. Namely, converting document coordinates to widget coordinates and clicking them. I though there wasn’t much value in providing the code of the failed attempts to use the Fill-and-Enclose tool.

But using a TabletMove event is bringing me a lot closer to my goal! However, for some reason, it does correctly drag the rectangle across the canvas, but when it releases, nothing happens.

In the meanwhile, I’ve also discovered QTest.mousePress and co., which does actually work as intended, as long as the canvas isn’t so zoomed in that it goes offscreen. See below for a video demonstration of both functions and their code.

from krita import Krita
from PyQt5.QtCore import *
from PyQt5.QtGui import *
from PyQt5.QtWidgets import *
from PyQt5.QtTest import QTest

def get_q_view(view):
    window = view.window()
    q_window = window.qwindow()
    q_stacked_widget = q_window.centralWidget()
    q_mdi_area = q_stacked_widget.findChild(QMdiArea)
    for v, q_mdi_view in zip(window.views(), q_mdi_area.subWindowList()):
        if v == view:
            return q_mdi_view.widget()

def get_q_canvas(q_view):
    scroll_area = q_view.findChild(QAbstractScrollArea)
    viewport = scroll_area.viewport()
    for child in viewport.children():
        cls_name = child.metaObject().className()
        if cls_name.startswith('Kis') and ('Canvas' in cls_name):
            return child

def get_transform(view):
    def _offset(scroller):
        mid = (scroller.minimum() + scroller.maximum()) / 2.0
        return -(scroller.value() - mid)
    canvas = view.canvas()
    document = view.document()
    q_view = get_q_view(view)
    area = q_view.findChild(QAbstractScrollArea)
    zoom = (canvas.zoomLevel() * 72.0) / document.resolution()
    transform = QTransform()
    transform.translate(
            _offset(area.horizontalScrollBar()),
            _offset(area.verticalScrollBar()))
    transform.rotate(canvas.rotation())
    transform.scale(zoom, zoom)
    return transform
    
def get_cursor_in_document_coords():
    app = Krita.instance()
    view = app.activeWindow().activeView()
    document = view.document()
    if document:
        q_view = get_q_view(view)
        q_canvas = get_q_canvas(q_view)    
        transform = get_transform(view)
        transform_inv, _ = transform.inverted()
        global_pos = QCursor.pos()
        local_pos = q_canvas.mapFromGlobal(global_pos)
        center = q_canvas.rect().center()
        inv_pos = transform_inv.map(local_pos - QPointF(center))
        centerCanvas = QPointF(0.5 * document.width(), 0.5 * document.height())
        return inv_pos + centerCanvas

def get_global_from_document_coords(doc_x, doc_y):
    app = Krita.instance()
    view = app.activeWindow().activeView()
    document = view.document()
    if document:
        q_view = get_q_view(view)
        q_canvas = get_q_canvas(q_view)
        transform = get_transform(view)
        
        # Calculate the center of the document
        centerCanvas = QPointF(0.5 * document.width(), 0.5 * document.height())
        
        #Convert document coordinates to coordinates relative to the center
        local_pos = QPointF(doc_x, doc_y) - centerCanvas
        
        # Apply the transformation to convert to canvas coordinates
        transformed_pos = transform.map(local_pos)
        
        # Find the center of the canvas
        center = q_canvas.rect().center()
        
        # Convert canvas coordinates to local widget coordinates
        widget_local_pos = transformed_pos + QPointF(center)

        return QPoint(int(widget_local_pos.x()), int(widget_local_pos.y()))


def sleep(value):
    loop = QEventLoop()
    QTimer.singleShot(value, loop.quit)
    loop.exec()

def click_canvas(x0, y0):
    """
    https://doc.qt.io/qt-5/qtabletevent.html
    """
    pos0 = get_global_from_document_coords(x0, y0)
    # pos1 = get_global_from_document_coords(x1, y1)
    app = Krita.instance()
    view = app.activeWindow().activeView()
    document = view.document()
    if document is None:
        return

    q_view = get_q_view(view)
    canvas = get_q_canvas(q_view)    

    
    device = QTabletEvent.Stylus
    pointer_type = QTabletEvent.Pen
    pressure = 1
    x_tilt = 0
    y_tilt = 0
    tangential_pressure = 0.0
    rotation = 0.0
    z_pos = 0
    key_state = Qt.NoModifier
    unique_id = 1234  # ???
    button = Qt.LeftButton
    buttons = Qt.LeftButton

    canvas.activateWindow()

    global_pos = QPointF(canvas.mapToGlobal(pos0))
    table_press = QTabletEvent(
        QEvent.TabletPress,
        pos0,
        global_pos,
        device,
        pointer_type,
        pressure,
        x_tilt,
        y_tilt,
        tangential_pressure,
        rotation,
        z_pos,
        key_state,
        unique_id,
        button,
        buttons)

    global_pos = QPointF(canvas.mapToGlobal(pos1))
    table_release = QTabletEvent(
        QEvent.TabletRelease,
        pos0,
        global_pos,
        device,
        pointer_type,
        pressure,
        x_tilt,
        y_tilt,
        tangential_pressure,
        rotation,
        z_pos,
        key_state,
        unique_id,
        button,
        buttons)

    QApplication.sendEvent(canvas, table_press)
    QApplication.sendEvent(canvas, table_release)

def drag_canvas_QTest(x0, y0, x1, y1):
    def sleep(value):
        loop = QEventLoop()
        QTimer.singleShot(value, loop.quit)
        loop.exec()
    pos0 = get_global_from_document_coords(x0, y0)
    pos1 = get_global_from_document_coords(x1, y1)
    app = Krita.instance()
    view = app.activeWindow().activeView()
    document = view.document()
    if document is None:
        return

    q_view = get_q_view(view)
    canvas = get_q_canvas(q_view)    
    
    device = QTabletEvent.Stylus
    pointer_type = QTabletEvent.Pen
    pressure = 1
    x_tilt = 0
    y_tilt = 0
    tangential_pressure = 0.0
    rotation = 0.0
    z_pos = 0
    key_state = Qt.NoModifier
    unique_id = 1234  # ???
    button = Qt.LeftButton
    buttons = Qt.LeftButton

    canvas.activateWindow()

    QTest.mousePress(canvas, Qt.LeftButton, Qt.NoModifier, pos0)
    sleep(500)
    QTest.mouseMove(canvas, pos1)
    sleep(500)
    QTest.mouseRelease(canvas, Qt.LeftButton, Qt.NoModifier, pos1)

def drag_canvas_TabletEvent(x0, y0, x1, y1):
    pos0 = get_global_from_document_coords(x0, y0)
    pos1 = get_global_from_document_coords(x1, y1)
    app = Krita.instance()
    view = app.activeWindow().activeView()
    document = view.document()
    if document is None:
        return

    q_view = get_q_view(view)
    canvas = get_q_canvas(q_view)    

    device = QTabletEvent.Stylus
    pointer_type = QTabletEvent.Pen
    
    global_pos = QPoint()
    pressure = 1
    x_tilt = 0
    y_tilt = 0
    tangential_pressure = 0.0
    rotation = 0.0
    z_pos = 0
    key_state = Qt.NoModifier
    unique_id = 1234  # ???
    button = Qt.LeftButton
    buttons = Qt.LeftButton

    canvas.activateWindow()
    
    table_press = QTabletEvent(
        QEvent.TabletPress,
        pos0,
        global_pos,
        device,
        pointer_type,
        pressure,
        x_tilt,
        y_tilt,
        tangential_pressure,
        rotation,
        z_pos,
        key_state,
        unique_id,
        button,
        buttons)

    table_move = QTabletEvent(
        QEvent.TabletMove,
        pos1,
        global_pos,
        device,
        pointer_type,
        pressure,
        x_tilt,
        y_tilt,
        tangential_pressure,
        rotation,
        z_pos,
        key_state,
        unique_id,
        button,
        buttons)
    
    table_release = QTabletEvent(
        QEvent.TabletRelease,
        pos1,
        global_pos,
        device,
        pointer_type,
        pressure,
        x_tilt,
        y_tilt,
        tangential_pressure,
        rotation,
        z_pos,
        key_state,
        unique_id,
        button,
        buttons)
    
    QApplication.sendEvent(canvas, table_press)
    sleep(500)
    QApplication.sendEvent(canvas, table_move)
    sleep(500)
    QApplication.sendEvent(canvas, table_release)

app = Krita.instance()
view = app.activeWindow().activeView()
document = view.document()

drag_canvas_TabletEvent(0,0,document.width(),document.height())
#drag_canvas_QTest(0,0,document.width(),document.height())

Krita.instance().activeDocument().refreshProjection()

If at all possible, I’d like to use the TabletEvent version, as the QTest function, as shown in the video, only works when the entire canvas is mostly on-screen, and if it isn’t it sends the mouse flying off somewhere, which is quite irritating.
Problem now is that I can’t for the life of me figure out why the TabletEvent function doesn’t trigger the Fill-and-Enclose tool on releasing.

[Citrus8684]

I hesitate to call it a solution, but the QTest function works well enough if you run the reset_display action first and move your pen away from the tablet.