use of external pip libraries in Python plugins

Hi

Few days ago I was a little bit irritated to read a comment concerning the difficulties -or impossibility- to use external libraries (particularly from pip) in Krita’s scripting.

And especially, the impression given by these words that the basic problem was a shameful choice on the part of the developers to privilege the current architecture (ie: publish Krita through appimage for Linux instead of usual distro package).

Maybe my english level doesn’t help and I misinterpreted the comment… :man_shrugging:

But ok, can’t stay with this without doing anything.

So, as I prefer use appimage because I think that’s the simplest way to ensure that:

  • All dependencies are up to date AND fully compatible with Krita
  • Krita is always up-to-date
  • Krita is available on all Linux without being dependent of distro specific package manager (rpm, apt, yum, … impossible to enumerate all and impossible for Krita’s team to build package for all possible package manager)

I took a look about how to execute pip within Krita’s appimage environment.

For curious, here results of my investigation.

Krita AppImage

In scripter, you can execute this
from PyQt5.Qt import *

import sys
import os
import runpy
import re


def pipInstallPath():
    """Return pip lib path
    
    Eventually:
    - create directory if not exist
    - add it to sys.path
    """
    returned=os.path.join(QStandardPaths.writableLocation(QStandardPaths.AppDataLocation), 'pykrita', 'piplib')
    
    if not os.path.isdir(returned):
        os.makedirs(returned)

    if not returned in sys.path:
        sys.path.append(returned)
        
    return returned


def pip(param):
    """Execute pip 
    
    Given `param` is a list of pip command line parameters 
    
    
    Example:
        To execute:
            "python -m pip install numpy"
        
        Call function:
            pip(["install", "numpy"])
    """
    def exitFct(exitCode):
        return exitCode
    
    pipLibPath=pipInstallPath()

    # keep pointer to original values
    sysArgv=sys.argv
    sysExit=sys.exit

    # replace exit function to be sure that pip won't stop script
    sys.exit=exitFct

    # prepare arguments for pip module
    sys.argv=["pip"] + param
    sys.argv.append(f'--target={pipLibPath}')
    
    #print("pip command:", sys.argv)
    runpy.run_module("pip", run_name='__main__')
    
    sys.exit=sysExit
    sys.argv=sysArgv


def checkPipLib(libNames):
    """Import a library installed from pip (example: numpy)
    
    If library doesn't exists, do pip installation
    """
    pipLibPath=pipInstallPath()
    
    if isinstance(libNames, list) or isinstance(libNames, tuple):
        installList=[]
        for libName in libNames:
            if isinstance(libName, dict):
                libNameCheck=list(libName.keys())[0]
                libInstall=libName[libNameCheck]
            elif isinstance(libName, str):
                libNameCheck=libName
                libInstall=libName
                
            try:
                # try to import module
                print("Try to load", libNameCheck)
                __import__(libNameCheck)
                print("Ok!")
            except Exception as e:
                print("Failed", str(e))
                installList.append(libInstall)

        if len(libInstall)>0:
            pip(["install"]+installList)
    elif isinstance(libNames, str):
        checkPipLib([libNames])

        
print('-------------------------------------------------------------------------')

checkPipLib([
            {"PIL": "Pillow"},       # module name (PIL) != pip installation name (Pillow)
            "numpy"                # module name = pip installation name
        ])

try:
    import numpy
    print("Numpy version", numpy.version.version)
except Exception as e:
    print("Can't import numpy", str(e))
    

try:
    import PIL
    print("PIL version", PIL.__version__)
except Exception as e:
    print("Can't import PIL", str(e))


print('-------------------------------------------------------------------------')

Results:

  • print("Numpy version", numpy.version.version)
    Numpy version 1.20.2
  • print("PIL version", PIL.__version__)
    PIL version 8.2.0

Modules are installed and ready to be used :slight_smile:
(note: I didn’t tested them yet, but I consider that if module is installed and loaded, that should be ok)

Scripter is interesting but it’s better to proceed installation of modules from a Plugin (finally it’s the target)
Provided code can easily be copied/pasted in a plugin:

  • If a required module is not installed, it will be installed at Krita’s startup (can be long according to your internet connection speed)
  • If a required module is already installed, installation will be just ignored
  • Modules installed from pip are located in ~/.local/share/krita/pykrita/piplib directory (so, it’s local to user’s Krita installation)

Krita for Windows

Ok for windows things are more tricky… I don’t why, but in appimage the pip module is available by default, but not with windows… (problem of license??)
Note: that my installed Python3 environment on windows (not related with Krita) don’t have pip module too!

So it need a little bit more of work…
First, I had to install pip on my Python3 environment
Once installed, I made a copy/paste of site-package directory (from my Python installation) to krita’s python directory.

Now, in scripter it doesn’t work: the only way I found to use pip is through a Plugin executed at Krita’s startup:


(yes, on Windows I still have a 4.4.2 :sweat_smile:)

And then in scripter:

Conclusion

It’s possible to install properly pip modules for Krita :slight_smile:

  • For appimage it’s easy, as everything is already available
  • For Windows, I don’t know: technically, it might be possible to zip binaries and provide a plugin ready to use that will install everything without complex action for user (clearly, what I did can’t be asked for a basic user) but I’m wondering about licenses (even under an open source license, providing pip windows binaries might be more complex than just providing source code… :thinking:)

Maybe some additional work have to be made to let the provided code being more generic and easier to use for plugins, but curently I don’t have time.
I keep this for the day I’ll need to use a pip library for a plugin…

Concerning pip module, maybe developpers can tell us more about why this library is available for appimage but has been excluded from windows binaries.
I’m curious :slight_smile:

Grum999

4 Likes

For the Windows ‘difficulties’, is there a difference between the installer package and the portable .zip package or is this purely a local Python plugin thing?

Hi

I usually use portable version in windows when possible, because I don’t really like to see softwares installing things in the registry and other places on which I have no control :slight_smile:

But I uncompressed installer and took a look on resources that are normally installed, in python directory the site-packages sub-directory content is empty (only a README like in portable version)

So I guess the same problem will occurs on an installed version.

On my side, the plugin part can easily be solved, just need some work to produce something that could be compatible for Windows and Linux AppImage.

The “blocking” part is pip binaries.
Don’t know why they’re missing in Krita and in python but it might be related to a license thing, because I simply copy/paste what I installed in a python environment into Krita’s Python directory (+fix sys.path to add new directory in list of directories’ modules).
If it’s “as simple as this” to install pip binaries with Krita, there’s might be a reason to not embed it with windows :man_shrugging:

Note, didn’t tried for MacOS because I have no possibility to test it :slight_smile:

Grum999

Krita doesn’t install anything in the registry, only in the install folder. Other than that it accesses AppData, but that’s something the portable Krita accesses as well.

Ah ok

Thank for info :slight_smile:

But you never know what a software will really do on system until you’ve installed it :sweat_smile:
So that’s a reflex to get a portable version when it’s available :wink:

After I’m only using my windows VM for testing Krita so I guess that I can try to pass over my habits :sweat_smile:

Grum999

1 Like

Most likely, it’s just because the Python we package with Krita has it, or doesn’t have it by default. There’s no conscious decision involved.

Just my random thoughts about the topic

What I have been doing to install extra resources on Blender per example is create a folder in my system Python > Blender and then do pip installations inside it. The thing is with Blender and Gimp or any other program that you can script for for that matter you have something that is the “Environment” that is independent of the OS installation folders visible frustration with Krita. My folders per example are on my portable drives. So I install Blender and just tell it where to find extra resources to be used so I don’t need the be reinstalling everything concerning PIP or place it on the OS system. Having to work on the system hidden folders … is just not good policy and then I am criticized every time for my weird workflow and saving habits when I talk about it.

Open source examples

Blender:

Gimp:

You can simply say where your Numpy is in your system you can use it in Blender or in Gimp like this.

I know this solution is not for the everyday user but it is by far tested and works and much needed when scripting is allowed. I have spoke of this before and no one really gave any importance to it which I find perplexing to this day.

Not to mention this solution would just mean redirecting Krita’s attention to another set of folders to try and find resources of a given type and not load them if they where not workable. It is not as hard as to try and fix it with PIP and possibly have redistribution issues. This would work in any system and with any appimage and people that are used to this already know what to do.

in Maya’s case it was a text file that held all the folder variables that redirected everything and remained on the hidden folders.

After I heard what he said on that thread honestly I agreed with him.

I get honestly a bit tired reading these threads. If people want an improved version of some feature in Krita, they should work on improving Krita, instead of complaining that something “is discouraged” into the void.

1 Like

I think you’re wrong and just don’t understand how all of this works.

Blender Appimage have exactly the same problem than Krita’s appimage: you can’t access to OS resources without problem because that’s an appimage.
Blender Appimage have a problem that Krita’s appimage don’t have: pip module is not available for blender appimage, so here it’s just impossible to use pip modules from Blender 's appimage

But as it’s Blender it’s better than Krita? :thinking:

Didn’t tried blender on windows, but I think the problem might be the same, I have to test…

Grum999

Blender appimage

Blender doesn’t distribute appimages, from what I can tell. They distribute native binaries.

Oh ok

As my Python for windows is not provided by default with pip module, this might be the same source :sweat_smile:

Thanks, in this case I’ll try to understand why python for windows is not provided with pip

Grum999

It’s not official maybe :man_shrugging:
But it exists :slight_smile:
And as it’s an appimage, even Blender can’t do more than appimage allows to do…

Grum999

Perhaps I’m misuderstanding, but I fail to see how this argument supports the conclusion… Regardless, that’s not an official appimage.

Your not following what I am trying to say then.

Blender does not have access to the system folder resources either. But you can use pip to manually install the resources in any folder. Installing those resources in a installation OS folder is not good idea.

I have done this before to get PyQt5 in Blender. But you need to access a folder where that Pip installation is present. Yes it is a manual installation and you can’t automate it by just telling Blender to get the dependencies but you will read them because they are properly installed. All my Blender installations have PyQt5 once I set up my environment no more work needed since it is all in my portable drive. It is literally plug and play now.

Just that problem is not on Krita’s side, but more about how appimage are working.

Even if it’s not an official appimage, Blender running in appimage have exactly the same problem than Krita concerning access to external resources.

Grum999

… But this doesn’t have relevance to @EyeOdin 's thoughts. The matter is about libraries installed into user folders, not system folders. I don’t see how appimages even have relevance in this discussion when EyeOdin is a Windows user…

Windows Python installation does provide Pip.

But you need to do add to the windows environment variables so Python works properly so it is not just install and your done. I do Pip installations per example now that it is running.

Because when I initiated this topic, it was about appimage problem and how to solve problem of external libraries with appimage.

My error was to also try to find a solution for windows users…
Next time, I’ll just take care of linux users to avoid discussion about windows, on which problem is different.

Grum999

Don’t know why, on my side it was not provided and I had to follow manual installation process of pip on windows…

Grum999

I think when your installing Python you need to check one of the boxes to have it right away.
Something like that, i know I messed it up the first time too.

And don’t worry if it works just for Linux there is no worry it still has validity still.

I just feel there are other paths to reach the same goal too that are more inclusive so you don’t waste your work time if that is not what your aiming for. Or might be worth considering I donno.

If I knew how to fix what I spoke this would have been done a good couple of months ago instead of me just talking, but I am barely a scripter let alone a developer to do anything at the moment. I think most people around forget I started learning Python just last year right after I arrived at this forum asking tons of stupid questions, I am a literal noob.