PATCHES ------- Dock-patches are a cheap way to create dock-compatible images, as opposed to downloading images with the functionality you want as large files - either as .tgz archives or by pulling them from Docker Hub. Instead, when you want a "Ruby 3" image or a "Python 2" image you just "apply" a patch named "ruby3" or "python2" to an existing container. Dock's patch tool helps you select, run and manage those patch-scripts with ease. It also offers you to commit the modified container under a different image name. A lot of useful patches are shipped with the main `dock` repository as a submodule, so take a look what's inside the $DOCK_PATH/patches directory. Or, better yet, run this command: $ dock -c patch --list Patches help `dock` achieve its eventual goal of being able to work with multiple container engines and not depend on any particular image format. Another advantage of this approach is that when running such patches against a customized container, your own changes are most likely to be preserved. TERMINOLOGY ----------- To better understand the rest of this document, it is necessary to introduce short but important terminology list, so as not to confuse seemingly similar things. But what is a patch-script? Let us quickly define it by describing what it may do. A patch script can be installing necessary packages as well removing them; it can be changing configuration files, creating new init.d scripts to run services, cleaning up logs, adding crontab tasks... You name it. Actually, since it's a Bash script, in can do anything you can do to your container manually - just faster - effectively turning it into a derivative image that you desire. * patch A patch is a collection of functions (it may be just one function, though, but it's kind of like arrays - an array with just one element is still an array), or code, if you may, that is intended to be executed inside the target container in order to modify it in some pre-defined way. * applying a patch Means a particular script will be run from inside the specified container of choice. The name of that container is passed to the main "patch tool" (see below) along with the names of the patches user (you?) wants to apply to the named container. * patch-function If patches are arrays, then patch-functions can be compared to array elements. They're actually numbered too, as you will soon see. * patch-scripts or patch-script files Those are files written in Bash script that contain all functions that may be executed on the target container. The "maybe" executed" statement means that each patch-script will be updated over time and newer functions will be added. Containers which have already had previous rounds of functions applied to them will ignore the older functions and only run those that have not yet been applied to it. * patch tool This is a separate script that resides in $DOCK_PATH/bin and may be invoked either directly or by calling `$ dock CONTAINER_NAME -c patch ...` command. This script is not going to be in your $PATH, so to run it directly, you'd need to actually cd into $DOCK_PATH/bin. Thus, the direct invocation is mean to be used more for debugging purposes. If you happen to invoke it directly, the first argument to it should be your target container name followed by an arbitrary number of arguments that are considered different "patch names". Normally, though, whether or not you're invoking it directly, you would usually be passing only one patch name, not multiple. USAGE ----- Patches are just Bash scripts located in $DOCK_PATH/patches - that's the default path for $DOCK_PATH/bin/patch executable to look for patch-scripts. To get a list of the available patch-scripts (and their descriptions) that ship along with the official `dock` toolset, run this command: $ dock -c patch --list The descriptions you'll see next to each patch's name will be "high-level", as if they were image descriptions. However, you may want to read description of each patch-function, which may be useful to assess whether you want to apply new patches. In that case, run this command, naming a particular patch-script that you're interested in: $ dock -c patch --list ruby3 For each patch-function that has a description (most should), it will be printed next to its name. For those patch-functions that lack one, only their names/indexes will be printed. If you are writing a patch-script or patch-functions always try to provide a description. It doesn't take a long time to write a short one-liner that helps users understand what your script as a whole or particular patch-function inside it is about to do to their container. For details on how to write your own patch scripts, see the WRITING PATCH-SCRIPTS section. You may also pass a container name, when running previous commands, in which case the output would be a bit more specific, only listing patches that are designed to be applied to the image your container is running. $ dock my_ruby3_container -c patch --list # POSSIBLE OUTPUT: # ubuntu20: provides dock-compatibility for the "ubuntu" image from Docker Hub # ruby3: rbenv & Ruby 3.1.1 for ubuntu20 image The output here shows you patch-scripts and their descriptions that have been applied to the container. It may not seem very useful, but it will give you information (specifically, patch-script names), which you can use to run this next command: $ dock my_ruby3_container -c patch --list ruby3 # POSSIBLE OUTPUT: # + Patch_1: install rbenv & ruby 3.1.1, updating dotfiles accordingly # - Patch 2: [no description] This output shows detailed information on patch-functions that belong to the named patch-script (ruby3) with relation to the container. We see that "ruby3" patch-script has two patch-functions so far. We also see that the first one has already been applied (the plus sign in front of the function indicates that), while the second patch function was not applied and it also doesn't have a description, so it's hard for us to tell what it does without looking inside the patch-script file (and that's why it is strongly recommended to always wrote descriptions for patch-functions). But suppose we trust it. Now we finally can get to the command which would apply all non-yet-applied patch-functions from the specified patch-script: # no need to add ".sh" to "ruby3", although you could $ dock CONTAINER_NAME -c patch ruby3 But suppose that `Patch_1` hasn't been applied either. In that case, you would've seen a different output from the `--list` command: $ dock my_ruby3_container -c patch --list ruby3 # POSSIBLE OUTPUT: # - Patch_1: install rbenv & ruby 3.1.1, updating dotfiles accordingly # - Patch 2: [no description] If the first patch-function has never been applied, then, of course, it will run first. But in that case, you also won't be seeing "ruby3" listed in output of the `dock my_ruby3_container -c patch --list` - and that's absolutely typical when you, for instance, want to create a new image from the base, dock-compatible image with some additional software installed. As stated, first and foremost, patches are a cheap way to create new images with specific things installed as opposed to downloading full images. Since official patches are placed inside its own repository - a submodule to the main repository - they get updates, which you may be interested in applying to your existing images and containers. Updates to patches don't always concern the target software for which your image may have been created, but they may be very crucial apply, because they might contain instructions to install security updates or perform some minor configuration tweaks. It is explained below in the INCREMENTAL PATCHING section exactly how those updates work (and how to disable them), but if you simply want to apply all of the updates released for images which are descendants of your personal image, then create/start a container based on that custom image of yours and then run this command: $ dock CONTAINER_NAME -c patch With no arguments provided, the patch tool will apply all updates which are compatible with with your image (to understand what is meant by compatibility here, please read the SELECTIVE & ANCESTRY PATCHING section. You may also hand-pick specific patches for specific ancestor-images and avoid all other updates. First, ask the patch tool to print all pending updates for the image your container is based on and carefully read the description for each patch: $ dock CONTAINER_NAME -c patch --list-updates NOTE: This will only currently work if you pull a new version of `dock` from the repository along with updating its submodules. Or alternatively, you can have DOCK/patches repository in some other location, which should then be added to $DOCK_PATCHES_PATH in ~/.dockrc. Until v1.0.0 is officially announced, updates to the main `dock` repository may make older versions of patch-scripts incompatible (although it is unlikely and will, be avoided, if possible). When all requested patches execute successfully, you'll be asked if you want to commit the patched container to an image with a new name. If you say yes, you will also be asked to allow the `cleanup_container` script to run before the container is committed to an image (the recommended answer is to agree) - see ./IMAGES.txt for details. In any case, all of these steps are automated and you only need to provide a little bit of information when the patch tool requests it by prompting you inside your terminal. INCREMENTAL PATCHING –––––––––––––––––––– When running a patch-function against a container, it won't be executed the second time - behaving much like a database migration. The information about which patch-functions from which patch-scripts have been applied is saved inside the container/image filesystem itself. The /usr/local/etc/applied_patches file holds information about names and last indexes of applied patches. The file would usually contain several lines with several different patch-script names. This is because the image from which your container had been derived is most likely to have some ancestors too and was itself created by the patching tool. Another reason for this file to have several lines with several patch-script names is that if you were to create an image that had both "nodejs16" and "ruby3" patches applied, the patch script would, obviously, place this information inside that file. Therefore, the file might end up containing something like this: ubuntu20 3 nodejs16 1 ruby3 2 SELECTIVE & ANCESTRY PATCHING ––––––––––––––––––––––––––––– Patches are normally meant for certain images with certain combination of software and configuration. This is why all patches in $DOCK_PATH/patches are placed inside directories named after image names they are meant for. But also, because image names change from the original image your container may be based on, you may want to pick shorter names such as "ruby3" instead of "ubuntu20-ruby3" when creating an image with a Ruby 3 installation. For that reason a /usr/local/etc/image_ancestry file exists inside each dock-compatible container. The patch tool uses it to track ancestor images patch-scripts updates. This file is updated automatically whenever you run a patch and create a new image from the modified container. IMPORTANT: Let this be repeated: /usr/local/etc/image_ancestry is only updated when you create an image with a new name using the dock's patch tool, but not when you update an existing image. NOTE: "Image name" here is meant for to the part after the forward slash, but before the semicolon character preceding tag names. That is to say, in this context, image name be "ubuntu20" if its full name is "dock/ubuntu20:stable". Having said that, the /usr/local/etc/image_ancestry file may look like this: ubuntu20 ruby3,nodejs16 These lines indicate to the dock's patch tool that it must check for three patch-scripts and then apply all new patch functions - in that very same order in which they are mentioned in the /usr/local/etc/image_ancestry file: from top-down, left-to-right. NOTE: In the example above, the reason there are two and not three lines and the second line lists two images instead of one is because in this particular example, we assume that when this image was created, both patches were applied to the container simultaneously - or at least before committing the container into a new image. This has certain implications discussed below. The /usr/local/etc/image_ancestry file may, in fact, be ignored, depending on the context: a) When no patch names are provided to the patch tool, it will fully rely on this file and apply the listed patches in the order they are listed. This happens when you want to apply new patches meant for the current container's image and also the patch updates meant for that image ancestors - essentially performing a full update. For, suppose we ran this command: $ dock sub.project.my -c patch Assuming that a container named "dock sub.project.my" exists, if the /usr/local/etc/image_ancestry file has lines naming particular images as ancestors of the image from which the container had been derived, the patch tool will attempt to apply new patch-functions from patch-scripts meant for each one of the images mentioned in the /usr/local/etc/image_ancestry file. If you DON'T ever want your image to receive updates from its ancestors, basically opting out of updates to some or all of ancestors or even the current image - it isn't a problem to that. Either of things listed below would achieve that purpose: * Before running the patch-tool, remove the undesired image names from container's /usr/local/etc/image_ancestry file. * Answer "n" or "no" when the patch tool prompts you and asks whether you wish to retain the direct ancestor of the image - it will prompt you after you pick the option to automatically create an image with a new name out of the modified container (and that prompt you'll see prior to the one discussed here, of course). b) When patch-script names are provided, as in this example: $ dock sub.project.my -c patch ruby3 The patch tool will ONLY run the named patches and will not be applying patches meant for container's image ancestors. Essentially, the /usr/local/etc/image_ancestry file will be fully ignored with regards to applying patches other then the ones specified in the command. NOTE: The /usr/local/etc/image_ancestry file will still be used to check whether the named patch-script exists for the image ancestors set and it will determine whether it can be applied to your container. For more information on how patch tool determines it, see the ANCESTRY RESOLUTION AND SUBDIRECTORIES IN PATCH PATHS section. TODO 1: The currently is no automatic mechanism to make you aware of the updates to the "DOCK/patches" repository, so keep an eye on it. TODO 2: There's also no automatic way to update all of your running containers that may benefit from an updated patch(s). WHEN TO USE PATCHES ------------------- Firstly, patches obviously only work if your image/container is running the same OS. You shouldn't be applying patches meant for Ubuntu to Alpine. This is also why inside the $DOCK/patches directory you'll see directories named after officially maintained dock-compatible images, which, in turn, contain patch-scripts themselves. However, you may apply multiple patch files to the same the container by providing multiple patch names when running the `dock -c patch` command. For instance, if you wanted an image with "Ruby 3" and "Node.js" installed, you may have run this command: $ dock CONTAINER_NAME -c ruby3 nodejs But how deep down the rabbit hole do you want to go with patching your containers and images? At some point it may be much easier, faster and reliable to create and distribute an image containing all the necessary the changes. Generally speaking, if your patch-file is a thousand lines long (that's a bit dramatic and perhaps you'd want to pull the plug earlier) consider just creating an image and making it publicly available to everyone or privately available to your team. NOTE: It's actually very much the same story with database migrations. At some point, when there are too many, you just reset it and have your database being bootstrapped from a schema file. And then you start adding new migrations on top of that (not that I'm a proponent of DB migrations, but I think it's fair comparison and the idea makes a lot more sense if we speak about container/image changes). WRITING PATCH-SCRIPTS --------------------- It is almost guaranteed that you will want to create images of your own, with your personal quirks and tweaks; or images for which the DOCK/patches repository doesn't have patch-scripts just yet. The first step, then, is to let dock patch tool aware of additional locations to look for patch-scripts. To do that, edit your ~/.dockrc file: find the line which contains the variable `$DOCK_PATCHES_PATH` and change its value much like you would change a `$PATH` variable in Bash. For instance, this is what the default looks like: DOCK_PATCHES_PATH="$DOCK_PATH/patches" You can add a few other locations using the colon character as a separator - and remember, that the locations listed first will also be the first ones where the dock's patch tool will perform its look-up for patch-scripts: DOCK_PATCHES_PATH="$HOME/d-patches:/home/mom/d-patches:$DOCK_PATH/patches" NAME CLASHES –––––––––––– By default name clashes are ignored and if there are identically named directories and/or files inside those directories, patch-scripts form both - given they are compatible - will be applied (again, following the order in which the paths are listed in $DOCK_PATCHES_PATH). However you may opt out of this default behavior and only apply patch-scripts from the first location where a patch script is found by changing the value of the variable in ~/.dockrc file: DOCK_PATCH_NAME_CLASH_BEHAVIOR="apply-first" # default was "apply-all" There's one more possible value for this variable that you can make use of. It's subtle, but may be marginally useful. Since each directory in $DOCK_PATCHES_PATH may contain patch-script files as well directories and even sub-directories with patch-script files [1], you may want to be specific as to what you would like to ignore when a clash happens. The following setting will ignore identically named directories that may appear in $DOCK_PATCHES_PATH locations and use the first patch-script found. DOCK_PATCH_NAME_CLASH_BEHAVIOR="apply-first-dir" However, if there's an identically named file that is found in another directory, which matches the container's ancestry [1], this option allows its invocation. [1] See ANCESTRY RESOLUTION AND SUBDIRECTORIES IN PATCH PATHS sub-section below for a thorough explanation. PATCH-SCRIPT STRUCTURE AND CONVENTIONS –––––––––––––––––––––––––––––––––––––– This section is useful if you're intending on writing your own patch files of modifying existing ones. Each patch-script must be a Bash script that contains at least one function which is named the same as the patch-script file containing it, except in CamelCase. For example, if your patch-script file name is `home_webserver.sh`, it must contain one function named `HomeWebserver()`. NOTE: The top-level function named after the patch-script file name (but in CamelCased) - is a way to imitate namespaces, which Bash doesn't have. Thus, function nesting is a way to avoid function name clashes, since users may be requesting to invoke multiple patch-scripts in one command, as has been demonstrated above. The top level function will have nested functions which shall be named starting with the word "Patch" followed by a underscore "_" and a number. The top level function MUST end with very specific line (marked in the code below). Here's an example of a custom patch-script: HomeWebserver() { Patch_1() { ... } Patch_2() { ... } Patch_3() { ... } --> source $DOCK_PATH/patches/run.sh } IMPORTANT: If you forget to add the source `$DOCK_PATH/patches/run.sh` or place it anywhere else but before the closing bracket of the main function, the patch functions either won't be invoked at all, or only the ones preceding this line will be invoked. These patch-functions are applied consecutively. But numbering doesn't have to be strictly incremental as in 1, 2, 3 etc. You may, for instance, use `yyyymmdd` format for the number if you're working within a relatively large team and/or there are a lot of frequent changes and merges in your version control system of choice. This kind numbering for patch-functions may help avoid unnecessary conflicts while merging, but for smaller teams it may be unnecessary. "I hear you said merging?"... Yes you should definitely keep your custom image patch-script directories under version control. I'd suggest one repository for all of your custom path-scripts or at least one repository for all path-scripts that are related in the sense that they are being used on images that are all a part of one large project. As mentioned previously, each dock-compatible container/image filesystem will have a file /usr/local/etc/applied_patches containing data about which patch-functions from which patch-scripts have been applied. Therefore, these functions will not run twice. If one of those patch-functions exits with an error, this file will not be updated and you can rerun the patch command or do whatever you need to do to the container manually and correct your patch script afterwards, so others can use it. Patch-scripts may also contain your own helper-functions as well standard ones provided by the DOCK/patches repository (the details about the standard helper-functions can be found a bit further down in this section). STANDARD HELPER FUNCTIONS ––––––––––––––––––––––––– If you want to make use of some standard helper functions, source the $DOCK_PATH/patches/helpers.sh at the very top of your patch-script file. You will then get access to a number of useful functions. In order not to pollute this documentation with the information on all of them, only a few examples will be given below. But you easily take a look at this file yourself and read comments, which explain things you may not be able to imply from a helper-function name. Here are the few examples: * `run_as USERNAME FUNCTION_NAME` Be default, all code inside the `Patch_N` functions is run as root. That's not always what you need. Sometimes you'd need large chunks of code that need to be run as another user and you'd want to avoid writing ugly multi-line functions wrapped up in `su USERNAME -c '...'`. This nice function does some meta-programming for you. You give it the custom helper-function name you wrote and it reads the body of that function with this nice `type` command [1], strips `function_name()`, the `{}` characters before and after the function body and other output artifacts. It then inserts the actual body of the function into the `su -c '...'` command. Just beware of variables and scopes. Why go through such trouble? Your own code readability. * `replace_welcome_message STRING` Takes a string and replaces the current welcome message in /usr/local/etc/welcome_message.sh file. You don't need to prepend each line with extra spaces or | characters - it will do it for you. This is how you'd typically use it inside one of the `Patch_N()` functions: local message="" message+="Welcome to Ruby 3.1.1 via rbenv (Ubuntu 20.04) container.\n" message+="This is a dock-compatible container." replace_welcome_message "$message" CUSTOM HELPER FUNCTIONS ––––––––––––––––––––––– To create your own helper-functions, place them either inside the top-level function or outside of it, but the recommended approach is to place them inside the top-level function unless you're absolutely sure there will be no name clashes. But, to if you were to think about it, had you wanted to share those helper-functions between several patch-scripts, it'd be a better idea to place them into a separate file and "source" that file from each patch-script that needs them. The naming convention for helper functions is to use snake_case names. [1] https://linuxcommand.org/lc3_man_pages/typeh.html or see `$ man type` CAN I USE OTHER PROGRAMMING LANGUAGES FOR PATCH-SCRIPTS? ––––––––––––––––––––––––––––––––––––––––––––------------ Not for scripts, yet, but since you can execute anything from inside a patch-function (it's normally run under the "root" user), it is indeed possible. I would not recommend doing this, because there's no real benefit, at least right now, to writing it in something like, say, Perl, or Ruby - with a patch-function written in Bash it's clear that you'd executing the exact same commands manually, form the container's command line. Perhaps, there will be a usecase in the future. PATCH-SCRIPT AND PATCH-FUNCTION DESCRIPTIONS –––––––––––––––––––––––––––––––––––––––––––– When other people are about to run your patch-script or specific functions in it, it is a good idea to give them an overview of what it is going to do. They can, of course, look inside your patch-script file and easily all the commands they'd be typing manually to install the software they need manually, but it's better to leave that option only for those who are concerned about security - and within an organization team(s), especially, there's an implied trust. Therefore, one only needs to know what the patch is about to do, not check whether it actually does what it says it would. [1] Whenever you create a patch-script of your own or add patch-functions you are strongly advised to include a short description for both the patch-script as a whole as well as its each individual function which name begins with `Patch_` (so we know it's not one of the helper functions, which don't need descriptions). To create a description for the patch-script define a local variable inside the top level function and assign a short string to it, with the description you think would be the best to explain your patch-script succinctly. For example: HomeWebServer() { local description='Standard ubuntu nginx installation form apt-get & an additional "http_fancyindex_module" to display directories content.' ... } You don't have to worry about extra white space: when the description is to be printed into a terminal the newlines will be rearranged properly and extra whitespace removed. Then you'd want to a description for each `Patch_N` function inside the `HomeWebServer()` function. This time though, you wouldn't create a local variable inside a patch-function - that wouldn't work. Instead create another variable inside the top-level function with a name that is the is the same as the name of the patch-function it's describing followed by "_description": HomeWebServer() { ... local Patch_1_description='Installs nginx and ngx_http_fancyindex_module with apt-get, creates an init.d script and adds "service start nginx" to /root/dock_bin/startup_jobs. Updates container welcome message.' Patch_1() { ... } } It doesn't matter where you define this variable, but, generally speaking, it's probably a good idea place it right before the definition of the patch-function it's describing. [1] Obviously, with the patches that ship with `dock` or some third-party patches you might want to take a look at the file. But here's a note to anyone who writes a patch-script or adds a patch function: don't write it so that it's hard to understand the code, because if anyone decides to inspect it is advantageous if it's simple, readable and short or, at least, not ridiculously long. ANCESTRY RESOLUTION AND SUBDIRECTORIES IN PATCH PATHS ––––––––––––––––––––––––––––––––––––––––––––––––––––– At first I thought "perhaps this is going too much" - in terms of features for the dock's patch tool... But I don't think it complicates anything for the end user, only for the patch-script author and even then - not too much. Whenever the patch tool checks $DOCK_PATCH_PATHS locations to pick the correct patch-script, it must verify whether this script is applicable for a particular image. It would've been possible, of course, to have a flat structure and name patch-script files "ubuntu20-ruby3.sh" instead of placing it under the ./ubuntu20 subdirectory and simply naming it ruby3.sh. But does it really change anything? Not at all. We still need to somehow make sure that any patch-script that a user is attempting to apply to the container is actually compatible with that container. How can we achieve this objective? Maybe let's just allow as many nested levels of subdirectories as we need. For instance, before we install "Ruby 3", we would want to install "rbenv" (which is one of Ruby's version managers). Rbenv is responsible for the installation of any particular Ruby version. Any while in reality for this particular case the DOCK/patches repository opted out for a flat structure, which installs both "rbenv" and "Ruby 3" using the same patch-script, it may be good example to illustrate how layers of patches can be applied and how the inner layer would always check whether the container it's attempting to modify had previously been patched by patches that are meant for the ancestors. This what the directory/file structure may look like if we were to separate "rbenv" from "Ruby 3" patches: ubuntu20 (DIR) ¯¯¯¯¯¯¯¯¯¯¯¯¯| | rbenv (DIR) ¯¯¯¯¯¯¯¯¯¯¯| |-- ruby2.sh |-- ruby3.sh If you attempt to run "ruby3" on a container based on an Arch Linux but no file arch/rbenv/ruby3.sh exists, then then the patch tool would simply inform you that the patch you are requesting isn't found. The patch tool determines container/image ancestry using the already mentioned /usr/local/etc/image_ancestry file and does not rely on any particular container-engine to extract that information. This helps `dock` achieve its eventual goal of being container-engine agnostic. Sometimes you would want to share part of the patch script meant for one image with another image - especially when the operating systems or distros of the images for which patches are intended are similar enough (as is the case with Debian & Ubuntu or any other Debian derivate). You can employ two techniques to avoid code duplication in such cases: 1) Symlinks Only use symlinks when you're absolutely sure that your patch-script will work in exactly the same way on another image you're targeting (this would probably be the case with Debian & Ubuntu, but not always). 2) Helper functions from helper scripts You may create helper scripts - those are also Bash files placed anywhere within the patches path dir structure (even at the top level) and then source them in the patch-scripts where wherever they're needed. This approach is useful when there are some similarities, but also quite a few differences between target images that would not allow a single patch-script to execute well on both. ON DETERMINISTIC BUILDS ----------------------- Ain't the objective.