Docker Slow Ext4 Partition Software
Docker Slow Ext4 Partition File
Docker Slow Ext4 Partition Tool

Been using Docker for Windows quite a lot lately.

Apr 16, 2017 Hence your host machine should have ext4 partition where docker is running and creating containers/ images from. $ sudo dockerd -storage-opt dm.basesize=50G. To let the changes take effect, you have to run the following commands: $ sudo service docker stop $ sudo rm -rf /var/lib/docker $ sudo service docker start. The docker zfs storage driver requires a zfs zpool to be created and mounted on the partition or disk where docker data should be stored. Snapshots (read-only) and Clones (read-write) are used to manage the images and containers. Zfs does not honor, or even allow, the dm.basesize setting.

Unfortunately running docker containers on windows is still not done natively, so we have to rely on a Linux Hyper-V Virtual Machine. This VM have a tendency to grow large quite fast if you work with many images, and yesterday I found myself running out of space on my C-drive due to that.

Luckily I thought I could just change the settings for the vhd location from the docker for windows settings to my other drive. But nope. That didn’t work. Then I tried to move the Hyper-V disk manually, and changing the path in Hyper-V, which just resulted in Docker for windows reprovisioning the Hyper-V machine to the C-drive again.

In the end, it turns out that this functionality is quite buggy, so the way I managed to move my disk was to close docker completely. And move the disk to my other drive E:VirtualMachineMobyLinuxVM.vhdx. When that is done you can edit
C:UsersblasiAppDataRoamingDockersettings.json and change the value of MobyVhdPathOverride to the new location of the VHDX E:VirtualMachineMobyLinuxVM.vhdx. When you restart the docker for windows service, it will pick up the change and change your Hyper-V configuration for you.

Categories: docker

Basic concepts¶

Adaptive Replacement Cache¶

Operating systems traditionally use RAM as a cache to avoid thenecessity of waiting on disk IO, which is comparatively extremelyslow. This concept is called page replacement. Until ZFS, virtuallyall filesystems used the Least Recently Used (LRU) page replacementalgorithm in which the least recently used pages are the first to bereplaced. Unfortunately, the LRU algorithm is vulnerable to cacheflushes, where a brief change in workload that occurs occasionallyremoves all frequently used data from cache. The Adaptive ReplacementCache (ARC) algorithm was implemented in ZFS to replace LRU. It solvesthis problem by maintaining four lists:

A list for recently cached entries.
A list for recently cached entries that have been accessed more thanonce.
A list for entries evicted from #1.
A list of entries evicited from #2.

Data is evicted from the first list while an effort is made to keepdata in the second list. In this way, ARC is able to outperform LRU byproviding a superior hit rate.

In addition, a dedicated cache device (typically an SSD) can be addedto the pool, with zpooladdpoolnamecachedevicename. The cachedevice is managed by the Level 2 ARC (L2ARC) which scans entries thatare next to be evicted and writes them to the cache device. The datastored in ARC and L2ARC can be controlled via the primarycache andsecondarycache ZFS properties respectively, which can be set onboth zvols and datasets. Possible settings are all, none, andmetadata. It is possible to improve performance when a zvol ordataset hosts an application that does its own caching by caching onlymetadata. One example is PostgreSQL. Another would be a virtualmachine using ZFS.

Alignment shift (ashift)¶

Top-level vdevs contain an internal property called ashift, whichstands for alignment shift. It is set at vdev creation and isotherwise immutable. It can be read using the zdb command. It iscalculated as the maximum base 2 logarithm of the physical sector sizeof any child vdev and it alters the disk format such that writes arealways done according to it. This makes 2^ashift the smallest possibleIO on a vdev. Configuring ashift correctly is important becausepartial sector writes incur a penalty where the sector must be readinto a buffer before it can be written.

ZFS will attempt to ensure proper alignment by extracting the physicalsector sizes from the disks. ZFS makes the implicit assumption thatthe sector size reported by drives is correct and calculates ashiftbased on that. The largest sector size will be used per top-level vdevto avoid partial sector modification overhead is eliminated. This willnot be correct when drives misreport their physical sector sizes.

In an ideal world, physical sector size is always reported correctlyand therefore, this requires no attention. Unfortunately, this is notthe case. The sector size on all storage devices was 512-bytes priorto the creation of flash-based solid state drives. Some operatingsystems, such as Windows XP, were written under this assumption andwill not function when drives report a different sector size.

Flash-based solid state drives came to market around 2007. Thesedevices report 512-byte sectors, but the actual flash pages, whichroughly correspond to sectors, are never 512-bytes. The early modelsused 4096-byte pages while the newer models have moved to an 8192-bytepage. In addition, “Advanced Format” hard drives have been createdwhich also use a 4096-byte sector size. Partial page writes sufferfrom similar performance degradation as partial sector writes. In somecases, the design of NAND-flash makes the performance degradation evenworse, but that is beyond the scope of this description.

Reporting the correct sector sizes is the responsibility the blockdevice layer. This unfortunately has made proper handling of devicesthat misreport drives different across different platforms. Therespective methods are as follows:

sd.conf on illumos
gnop on FreeBSD
-oashift= on Linux and potentially other OpenZFSimplementations

``-o ashift=` is convenient, but it is flawed in that the creation ofpools containing top level vdevs that have multiple optimal sectorsizes require the use of multiple commands. A newer syntax that willrely on the actual sector sizes has been discussed as a cross platformreplacement and will likely be implemented in the future.

In addition, a database of drives known to misreport sector sizes isused to automatically adjust ashift without the assistance of thesystem administrator. This approach is unable to fully compensate formisreported sector sizes whenever drive identifiers are usedambiguously (e.g. virtual machines, iSCSI LUNs, some rare SSDs), butit does a great amount of good. The format is roughly compatible withillumos’ sd.conf, and it is expected that other implementationswill integrate the database in future releases. Strictly speaking,this database does not belong in ZFS, but the difficulty of patchingthe Linux kernel (especially older ones) necessitated that this beimplemented in ZFS itself for Linux. The same is true forMacZFS. However, FreeBSD and illumos are both able to implement thisin the correct layer.

Compression¶

Internally, ZFS allocates data using multiples of the device’s sectorsize, typically either 512 bytes or 4KB (see above). When compressionis enabled, a smaller number of sectors can be allocated for eachblock. The uncompressed block size is set by the recordsize filesystem property (defaults to 128KB) or the volblocksize volumeproperty (defaults to 8KB).

The following compression algorithms are available:

LZ4: New algorithm added after feature flags were created. It issignificantly superior to LZJB in all metrics tested. It is newdefault compression algorithm (compression=on) in OpenZFS[1], butnot all platforms have adopted the commit changing it yet.
LZJB: Original default compression algorithm (compression=on) for ZFS. Itwas created to satisfy the desire for a compression algorithmsuitable for use in filesystems. Specifically, that it providesfair compression, has a high compression speed,

Docker Slow Ext4 Partition